Apparatus and method of safety support for vehicle

ABSTRACT

A vehicle safety support apparatus includes: a driver monitoring sensor configured to monitor a driver; an external environment monitoring sensor configured to monitor an external environment of a vehicle; a driver input filtering unit configured to filter a vehicle control input from the driver; and at least one processor connected to the driver monitoring sensor, the external environment monitoring sensor, and the driver input filtering unit, the at least one processor configured to: determine criterion based on data acquired from the driver monitoring sensor and the external environment monitoring sensor; determine whether to take over a driving control of the vehicle in response to the data acquired from the driver monitoring sensor and the external environment monitoring sensor meeting the criterion; and perform autonomous driving to move the vehicle to a safe area in response to determining to take over the driving control from the driver.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application claims priority from and the benefit of U.S.patent application Ser. No. 15/639,843, filed on Jun. 30, 2017, to beissued as U.S. Pat. No. 10,446,031, which claims priority from U.S.Provisional Application No. 62/471,114, filed on Mar. 14, 2017, each ofwhich is incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

The present invention relates to apparatus and method of vehicle safetysupport, and more particularly, to a vehicle safety support apparatusand method which is capable of supporting the safety of a driver usingan autonomous driving system.

In general, a vehicle provides movement convenience and time efficiencyto people, but a driver may cause much damage to surrounding people aswell as the driver, due to the driver's carelessness. Therefore,attention is required for using the vehicle. In particular, the recenttechnological convergence between vehicles and ICT (Information &Communication Technology) has made the vehicles intelligent andadvanced. Thus, a safe driving support system installed in each vehiclerecognizes a dangerous situation, and informs a driver of the dangeroussituation.

The conventional safe driving support system in the vehicle recognizesthe dangerous situation mainly by collecting information throughexternal sensors such as a radar and camera and determining an accidentrisk such as a lane departure or collision. Furthermore, the safedriving support system informs the driver of the dangerous situation bydisplaying the dangerous situation on a display device (for example,flickering a warning light) or outputting a voice. However, the voiceoutputted by the safe driving support system may be buried in noisecaused by high-speed driving, or the warning light may not be visuallyand auditorily recognized when the driver keeps eyes forward whilefocusing his attention on the driving or dozes off at the wheel.

The research of IIHS in the US has also concluded that the lanedeparture warning and assist system is not enough to prevent a roaddeparture accident or the like.

The related art is disclosed in Korean Patent Registration No.10-0282903 published on Dec. 2, 2000.

As described above, the conventional safety support systems provide thefunction of warning a driver of drowsy driving, careless driving orcollision risk. However, when the driver does not respond to the warningor cannot normally perform driving, the possibility that an accidentwill occur inevitably increases.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed to a vehicle safetysupport apparatus and method which is capable of preventing an accidentby moving a vehicle to a safe area when a driver does not normally drivea vehicle or cannot control the vehicle any more.

In one embodiment, a vehicle safety support apparatus may include: adriver monitoring unit configured to monitor a driver; an externalenvironment monitoring unit configured to monitor an externalenvironment of a vehicle; and a control unit configured to determine adriving control for the vehicle based on data acquired from the drivermonitoring unit and the external environment monitoring unit, andperform autonomous driving to move the vehicle to a safe area, whendetermining to take over the driving control from the driver.

The control unit may estimate a driver availability based on the dataacquired from the driver monitoring unit, estimate a traffic hazardbased on the data acquired from the external environment monitoringunit, and determine the driving control based on the estimated driveravailability and traffic hazard.

The control unit may determine to take over the driving control from thedriver when the vehicle is in immediate hazard situation and there is noresponse by the driver.

The control unit may perform autonomous driving to get out of theimmediate hazard situation.

The vehicle safety support apparatus may further include a driver inputfiltering unit configured to filter a vehicle control input by thedriver. When determining to take over the driving control from thedriver, the control unit may control the driver input filtering unit toblock the vehicle control input by the driver.

The control unit may operate a hazard lamp of the vehicle whenperforming the autonomous driving.

The control unit may transmit a signal calling for a help through acommunication unit, after moving the vehicle to the safe area.

The control unit may perform the autonomous driving based on the dataacquired from the external environment monitoring unit.

The driver monitoring unit may include one or more of a camera forfilming the driver, a steering wheel angle sensor, an accelerator pedalsensor and a brake pedal sensor.

The external environment monitoring unit may include one or more of acamera, radar and ultrasonic sensor, which detect the outside of thevehicle.

In another embodiment, a vehicle safety support method may include:monitoring, by a control unit, a driver and an external environment of avehicle; estimating, by the control unit, a driver availability andtraffic hazard based on data acquired in the monitoring of the driverand the external environment; determining, by the control unit, adriving control based on the estimated driver availability and traffichazard; and performing, by the control unit, autonomous driving to movethe vehicle to a safe area, when determining to take over the drivingcontrol from the driver.

The vehicle safety support method may further include blocking, by thecontrol unit, a vehicle control input by the driver, when determining totake over the driving control from the driver.

The vehicle safety support method may further include transmitting, bythe control unit, a signal calling for a help through a communicationunit, after the performing of the autonomous driving to move the vehicleto the safe area.

In the monitoring of the driver and the external environment, thecontrol unit may monitor one or more of the physical feature, postureand control intention of the driver.

In the monitoring of the driver and the external environment, thecontrol unit may monitor the road and traffic environment outside thevehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of a vehiclesafety support apparatus in accordance with an embodiment of the presentinvention.

FIGS. 2 to 4 are diagrams for describing the technical concept of thevehicle safety support apparatus in accordance with the embodiment ofthe present invention.

FIG. 5 is a flowchart for describing a vehicle safety support method inaccordance with an embodiment of the present invention.

FIG. 6 is a block diagram illustrating a system for managing a dangerousdriving index for vehicles, according to an embodiment of the presentinvention.

FIG. 7 is a block diagram illustrating a detailed configuration of anambient environment recognizer illustrated in FIG. 6.

FIG. 8 is a diagram schematically illustrating three dangerous sectionswhich are divided by a peripheral vehicle load calculator of FIG. 7,based on a time to collision (TTC).

FIG. 9 is a diagram illustrating a configuration of a sensor fordetecting peripheral vehicles located in each of the dangerous sectionsof FIG. 8.

FIG. 10 is a diagram illustrating an example of detecting peripheralvehicles by using the sensor illustrated in FIG. 9.

FIG. 11 is a flowchart illustrating a method of managing a dangerousdriving index for vehicles, according to an embodiment of the presentinvention.

FIG. 12 is a block diagram for describing another embodiment of theambient environment recognizer illustrated in FIG. 7.

FIG. 13 is a block diagram illustrating a detailed configuration of adetection section generator illustrated in FIG. 12.

FIG. 14 is a flowchart illustrating a method of optimizing a detectionsection, according to an embodiment of the present invention.

FIG. 15 is a block diagram illustrating a detailed configuration of adriver status sensing system illustrated in FIG. 6.

FIG. 16 is a diagram illustrating an apparatus for pieces of acquiredinformation, according to an embodiment of the present invention.

FIG. 17 is a diagram for describing a method of checking the eye closingof a driver, according to an embodiment of the present invention.

FIGS. 18A to 18F are a graph for describing a drowsiness load of adriver in a driving interruption load according to an embodiment of thepresent invention.

FIG. 19 is a flowchart illustrating an operation of checking the eyeclosing of a driver, according to an embodiment of the presentinvention.

FIG. 20 are a diagram for describing an observation negligence load of adriver in the driving interruption load according to an embodiment ofthe present invention.

FIG. 21 is a diagram illustrating an output of a screen according to anembodiment of the present invention.

FIG. 22 is a flowchart illustrating a driver status sensing methodaccording to an embodiment of the present invention.

FIG. 23 is a block diagram schematically illustrating a situationdetection apparatus according to an embodiment of the present invention.

FIG. 24 is an illustrative view of a driver status detection section.

FIG. 25 is an illustrative view of a vehicle surrounding situationdetection section.

FIG. 26 is a block diagram illustrating a determination unit.

FIG. 27 is a block diagram illustrating a warning unit.

FIG. 28 is a flowchart schematically illustrating a situation detectionmethod according to another embodiment of the present invention.

FIG. 29 is a flowchart illustrating a driving pattern learning step.

FIG. 30 is a view illustrating a state in a weighting determinationstep.

FIG. 31 is a flowchart illustrating a calculation learning step.

FIG. 32 is a flowchart illustrating an examination step.

FIG. 33 is a flowchart illustrating a warning step.

FIGS. 34 to 36 are detailed flowcharts illustrating the situationdetection method.

FIG. 37 is a block diagram illustrating a configuration of an apparatusfor detecting a driver status according to an embodiment of the presentinvention.

FIG. 38 is a view schematically illustrating a configuration of aninformation acquisition unit according to the embodiment of the presentinvention.

FIG. 39 is an exemplified view illustrating an ECG sensor and a PPGsensor according to the embodiment of the present invention.

FIG. 40 is an exemplified view illustrating an EEG sensor according tothe embodiment of the present invention.

FIG. 41 is an exemplified view illustrating a driving load displaydevice according to the embodiment of the present invention.

FIG. 42 is a flowchart schematically illustrating a method of detectinga driver status according to another embodiment of the presentinvention.

FIGS. 43 and 44 are flowcharts illustrating an information acquisitionstep in the method of detecting a driver status according to theembodiment of the present invention.

FIGS. 45 and 46 are flowcharts illustrating a calculation step in themethod of detecting a driver status according to the embodiment of thepresent invention.

FIG. 47 is a flowchart illustrating a first warning step in the methodof detecting a driver status according to the embodiment of the presentinvention.

FIGS. 48 and 49 are flowcharts illustrating the method of detecting adriver status according to the embodiment of the present invention.

FIG. 50 is a view for explaining a method of determining that a drivercloses eyes in the method of detecting a driver status according to theembodiment of the present invention.

FIG. 51 is a view for explaining a visible range during no-load drivingdepending on a wheel angle in the method of detecting a driver statusaccording to the embodiment of the present invention.

FIGS. 52 and 53 are views for explaining a method of determining adriver's viewing range in the method of detecting a driver statusaccording to the embodiment of the present invention.

FIG. 54 is a flowchart schematically illustrating a method of detectinga driver status which includes a driver status determination steputilizing an ECG according to still another embodiment of the presentinvention.

FIG. 55 is a detailed flowchart illustrating the driver statusdetermination step utilizing the ECG according to the embodiment of thepresent invention.

FIGS. 56 and 57 are views for explaining a method of determining adriver status from a driver's heart distribution chart and hearthistogram in the method of detecting a driver status according to theembodiment of the present invention.

FIG. 58 is a flowchart schematically illustrating a method of detectinga driver status which includes a driver status determination steputilizing an EEG according to yet another embodiment of the presentinvention.

FIGS. 59 and 60 are detailed flowcharts illustrating the driver statusdetermination step utilizing the EEG according to the embodiment of thepresent invention.

FIG. 61 is a view for schematically explaining a method of determining adriver status utilizing the EEG in the method of detecting a driverstatus according to the embodiment of the present invention.

FIG. 62 is a table illustrating a frequency range and characteristic ofeach brainwave.

FIG. 63 is a diagram for explaining a method of finding a frequencyrange for each brainwave using a Bayesian network.

FIG. 64 is a conceptual diagram illustrating a driver status deductionstep using the Bayesian network according to the embodiment of thepresent invention.

FIGS. 65 to 68 are detailed flowcharts illustrating a method ofdetermining a driver status utilizing an ECG and an EEG according to afurther embodiment of the present invention.

FIG. 69 is diagrams for describing a basic algorithm of the vehiclesafety support apparatus in accordance with the embodiment of thepresent invention.

DESCRIPTION OF SPECIFIC EMBODIMENTS

Embodiments of the invention will hereinafter be described in detailwith reference to the accompanying drawings. It should be noted that thedrawings are not to precise scale and may be exaggerated in thickness oflines or sizes of components for descriptive convenience and clarityonly. Furthermore, the terms as used herein are defined by takingfunctions of the invention into account and can be changed according tothe custom or intention of users or operators. Therefore, definition ofthe terms should be made according to the overall disclosures set forthherein.

FIG. 1 is a block diagram illustrating the configuration of a vehiclesafety support apparatus in accordance with an embodiment of the presentinvention.

As illustrated in FIG. 1, the vehicle safety support apparatus inaccordance with the embodiment of the present invention may include acontrol unit 100, a driver monitoring unit 110, an external environmentmonitoring unit 120, a communication unit 130, a GPS 140 and a driverinput filtering unit 150. The control unit 100 may include a driveravailability estimation unit 101, a traffic hazard estimation unit 102,a driving control determination unit 103 and an autonomous drivingcontrol unit 104.

The control unit 100 may be implemented with a processor such as an ECU

(Electronic Control Unit). At this time, the control unit 100 mayinclude a plurality of processors. That is, each of the driveravailability estimation unit 101, the traffic hazard estimation unit102, the driving control determination unit 103 and the autonomousdriving control unit 104 may be implemented as an individual processor.Alternatively, the functions of the control unit 100 may be distributedto one or more processors. For example, the driver availabilityestimation unit 101 and the traffic hazard estimation unit 102 may beimplemented on a single processor. In this case, each of the functionsof the driver availability estimation unit 101, the traffic hazardestimation unit 102, the driving control determination unit 103 and theautonomous driving control unit 104 may be configured in the form of acontrol algorithm or logic.

Also, the functions of the control unit 100 may be implemented as a partof another system mounted in the vehicle, for example, ADAS (AdvancedDriver Assistance System) or LDWS (Lane Departure Warning System).

The driver monitoring unit 110 may monitor the physical features,physical characteristics, posture and control intention of a driver. Thedriver monitoring unit 110 may include various sensors and devices forperforming the above-described function. For example, the drivermonitoring unit 110 may include a camera or radar for monitoring thephysical features, physical characteristics and posture of the driver,and a steering wheel angle sensor, acceleration pedal sensor and brakepedal sensor for monitoring the vehicle control intention of the driver.

In addition, the driver monitoring unit 110 may include a suspensionmotion recognition sensor, multi-function maneuver recognition sensor,voice recognition sensor, AVN maneuver recognition sensor, airconditioning device maneuver recognition sensor, gear box sensor,console box maneuver recognition sensor, glove box maneuver recognitionsensor, wearable sensors such as ECG (Electrocardiogram) sensor, EEG(Electroencephalogram) sensor and PPG (Photoplethysmography) sensor, andmonitor the eyelids of the driver, the pupils of the driver, the speedof a steering wheel, the angle of the steering wheel, a motion of thesuspension, whether the accelerator pedal is maneuvered, whether thebrake pedal is maneuvered, whether the multi-function is maneuvered,whether the driver is talking, whether the AVN is maneuvered, whetherthe air conditioning device is maneuvered, whether the gear box ismaneuvered, whether the console box is maneuvered, whether the glove boxis maneuvered, the ECG of the driver, the EEG of the driver and thelike.

The external environment monitoring unit 120 may monitor the road andtraffic environment outside the vehicle. The external environmentmonitoring unit 120 may include various sensors and devices forperforming the above-described function. For example, the externalenvironment monitoring unit 120 may include a camera, radar, ultrasonicsensor and the like, in order to monitor the road and trafficenvironment outside the vehicle. At this time, an AVM (Around View Mode)device or black box for a vehicle may be employed as the camera includedin the external environment monitoring unit 120, and the radar orultrasonic sensor included in the ADAS or LDWS mounted in the vehiclemay be employed as the sensors and devices included in the externalenvironment monitoring unit 120.

The communication unit 130 may communicate with the outside of thevehicle. For example, the communication unit 130 may be configured touse a mobile communication network, near field communication or acommunication platform for a vehicle. The mobile communication networkmay include CDMA (Code Division Multiple Access), GSM (Global System forMobile communications), WCDMA (Wideband Code Division Multiple Access),LTE (Long Term Evolution) and WiBro (Wireless Broadband), the near fieldcommunication may include Wi-Fi, and the communication platform for avehicle may include TPEG (Transport Protocol Expert Group), TIM (TrafficInformation System), RDS (Radio Data System), telematics for a vehicle,DSRC (Dedicated Short Range Communication), WAVE (Wireless Access inVehicular Environment).

The GPS 140 may receive the positional information of the vehicle. Thatis, the GPS 140 may receive information on the position of the vehicle,using a satellite navigation system.

The driver input filtering unit 150 may filter a vehicle control inputby the driver, for example, a steering wheel maneuver, accelerationpedal maneuver, brake pedal maneuver or the like. That is, the driverinput filtering unit 150 may block a vehicle control input by the driverfrom being transmitted to devices in the vehicle (a steering 200, abrake 300, a hazard lamp 400 and the like).

For example, when a steering wheel maneuver of the driver is performedby an electronic control system, for example, MDPS (Motor Driven PowerSteering) or EPS (Electric Power Steering), the driver input filteringunit 150 may be implemented with an ECU capable of transmitting acontrol command to the electronic control system. At this time, thedriver input filtering unit 150 may be implemented as a separateprocessor, but implemented as control logic in the processorconstituting the control unit 100.

Alternatively, when a steering wheel maneuver of the driver is performedby a mechanical connection, for example, a hydraulic power steering, thedriver input filtering unit 150 may include a component for blocking atransmission of oil pressure.

As such, the driver input filtering unit 150 may be implemented invarious forms depending on the design specification of the vehicle.However, since the specific configurations can be implemented throughproper design changes by a person skilled in the art, the detaileddescriptions thereof are omitted herein.

The control unit 100 may control the operations of the respective unitsof the vehicle safety support apparatus in accordance with theembodiment of the present invention. That is, the control unit 100 maycontrol the driver monitoring unit 110 to monitor the physical features,posture and control intention of the driver, control the externalenvironment monitoring unit 120 to monitor the road and trafficenvironment outside the vehicle, control the communication unit 130 toperform data transmission/reception, control the GPS 140 to acquire theposition information of the vehicle, and control the filtering operationof the driver input filtering unit 150.

The driver availability estimation unit 101 may estimate driveravailability based on the physical features (characteristics), postureand control intention of the driver, acquired through the drivermonitoring unit 110.

The driver availability may indicate whether the driver can effectivelycontrol the vehicle. For example, the driver availability may bedesigned to have a value of 0 to 1. When the driver availability isclose to 1, it may indicate that the driver can effectively control thevehicle, and when the driver availability is close to 0, it may indicatethat the driver cannot effectively control the vehicle. However, sincethe present embodiment is not limited to such a design, a variety ofdesign methods may be employed, which indicate whether the driver caneffectively vehicle control.

The driver availability estimation unit 101 may calculate the driveravailability by inputting the data acquired through the drivermonitoring unit 110 to a preset algorithm. For example, the driveravailability estimation unit 101 may determine whether the driver isdozing at the wheel, by considering the size of the driver's pupil orthe driver's posture. When it is determined that the driver is dozing atthe wheel, the driver availability may be estimated to a value close to0.

Furthermore, the driver availability estimation unit 101 may acquireheartbeat information as the physical features of the driver, andcalculate the driver availability by determining an abnormal state ofthe driver based on the heartbeat information or determining whether thedriver abnormally performs driving control, based on the angle of thesteering wheel or the extent that the accelerator pedal or brake pedalis stepped on.

In addition, various types of algorithms for estimating the driveravailability may be applied. For example, a dangerous driving index, anintegrated risk index of a driver, a driving load and the like may becombined to estimate the driver availability. In other embodimentsdescribed later, a variety of methods will be described.

The traffic hazard estimation unit 102 may estimate a traffic hazardbased on the road and traffic environment outside the vehicle, throughthe external environment monitoring unit 120.

The traffic hazard may indicate the possibility that an accident willoccur. For example, the traffic hazard may be designed to have a valueof 0 to 1. When the traffic hazard is close to 1, it may indicate thatthe possibility of accident occurrence is high, and when the traffichazard is close to 0, it may indicate that the possibility of accidentoccurrence is low. However, since the present embodiment is not limitedto such a design, various design methods indicating the possibility ofaccident occurrence can be employed.

The traffic hazard estimation unit 102 may calculate the traffic hazardby inputting the data acquired through the external environmentmonitoring unit 120 to a preset algorithm. For example, the traffichazard estimation unit 102 may detect an obstacle, detect a VRU(Vulnerable Road User), detect a lane, recognize a vehicle, recognize aroad edge, and detect road surface friction, using the camera, radar,ultrasonic sensor and the like. Then, the traffic hazard estimation unit102 may calculate the possibility of accident occurrence by collectivelyanalyzing the detection results.

The traffic hazard estimation of the traffic hazard estimation unit 102may be performed through an algorithm used by a conventional collisionwarning system. In other embodiments described later, a variety ofmethods will be described.

The driving control determination unit 103 may determine a drivingcontrol based on the estimated driver availability and traffic hazard.That is, the driving control determination unit 103 may determinewhether to allow the driver to continuously drive the vehicle or to takeover the driving control from the driver in order to perform autonomousdriving.

The driving control determination unit 103 may determine the drivingcontrol by applying the estimated driver availability and traffic hazardto a preset algorithm. For example, when the estimated driveravailability is equal to or less than a high threshold value or theestimated traffic hazard is equal to or more than a low threshold value,the driving control determination unit 103 may determine to take overthe driving control for the vehicle from the driver in order to performautonomous driving, or determine the driving control for the vehicledepending on a result obtained by combining the estimated driveravailability and the estimated traffic hazard. However, since thepresent embodiment is not limited to such an algorithm, various methodsfor determining the driving control for the vehicle may be used.

When the driving control determination unit 103 determined to take overthe driving control for the vehicle from the driver in order to performautonomous driving, the autonomous driving control unit 104 may controlthe respective devices in the vehicle (for example, the steering 200,the brake 300, the emergency light 400 and the like) to performautonomous driving.

When the driver does not normally drive the vehicle or cannot controlthe vehicle any more, the autonomous driving control unit 104 may movethe vehicle to a safe area through autonomous driving, in order toprotect the driver by preventing an accident.

For this operation, the autonomous driving control unit 104 mayestablish a driving plan, and perform driving control to move thevehicle to the safe area. The driving plan may include checking a closesafe area, setting a path to the safe area, planning to change a line,performing road exit control, performing collision avoidance control,and performing emergency driving, and the driving control may includelane following, lane changing, pulling off the road and holding.

At this time, the autonomous driving control unit 104 may performautonomous driving by utilizing the information acquired through theexternal environment monitoring unit 120, the position informationacquired through the GPS 140, the map information and the like, orperform autonomous driving by utilizing various techniques related toautonomous driving.

The safe area to which the vehicle will be moved may be included in themap information.

When the autonomous driving control unit 104 performs autonomousdriving, the control unit 100 may control the driver input filteringunit 150 to block vehicle control by the driver. Then, the vehicle canbe safely moved only through autonomous driving.

During the autonomous driving of the vehicle, the autonomous drivingcontrol unit 104 may operate the hazard lamp 400 of the vehicle in orderto notify emergency driving.

When the vehicle is positioned in the safe area after exiting from theroad, the autonomous driving control unit 104 may hold the vehicle, andthe control unit 100 may call for a help through the communication unit130. For example, a message indicating the emergency situation may beautonomously transmitted to 911.

In one embodiment, the control unit 100 may determine to take over thedriving control from the driver when the vehicle is in immediate hazardsituation and there is no response by the driver.

A lane departure situation, an intersection approach situation, acollision with another vehicle situation may be an example of theimmediate hazard situation. The criterion for determining the immediatehazard situation is preset in the control unit 100. And algorithms, usedin the traffic hazard estimating of the subject invention, aconventional lane departure detection system, a conventional collisionwarning system, may be used as the determination criterion. The controlunit 100 may determine whether the vehicle is in intersection approachsituation by using GPS and map information.

The control unit 100 may determine that there is no response by thedriver when a vehicle control of corresponding to each of the immediatehazard situation is not performed by the driver. For example, returningvehicle to lane, emergency braking to avoid another vehicle orintersection running through should be performed according to each ofthe immediate hazard situation.

Therefore, the control unit 100 takes over the driving control from thedriver and performs the vehicle control of corresponding to each of theimmediate hazard situation to get out of the immediate hazard situationwhen the vehicle is in immediate hazard situation and there is noresponse by the driver. That is, the vehicle can do “recovery” or“rescue” maneuver to eliminate the immediate hazard.

FIGS. 2 to 4 are diagrams for describing the technical concept of thevehicle safety support apparatus in accordance with the embodiment ofthe present invention.

As shown in FIGS. 2 to 4, the vehicle safety support apparatus inaccordance with the embodiment of the present invention may monitor thedriver availability by detecting various data and applying the data tothe algorithms. When determining that a risk of accident is presentwhile the driving control by the driver is maintained, the vehiclesafety support apparatus may filter an input of the driver, and move thevehicle to a safe area through autonomous driving, thereby preventing avehicle accident.

FIG. 5 is a flowchart for describing a vehicle safety support method inaccordance with an embodiment of the present invention.

As illustrated in FIG. 5, the control unit 100 may monitor a driver andthe external environment at step S10. That is, the control unit 100 maymonitor the physical features, posture and control intention of thedriver through the driver monitoring unit 110, and monitor the road andtraffic environment outside the vehicle through the external environmentmonitoring unit 120.

Then, the control unit 100 may estimate a driver availability andtraffic hazard according to the monitoring results of step S10, at stepS20. That is, the control unit 100 may estimate the driver availabilitybased on the physical features, posture and control intention of thedriver, acquired through step S10, and estimate the traffic hazard basedon the road and traffic environment outside the vehicle.

After step S20, the control unit 100 may determine a driving controlbased on the driver availability and traffic hazard estimated at stepS20, at step S30. That is, the control unit 100 may determine whether toallow the driver to continuously drive the vehicle according to thecontrol of the driver or to take over the driving control from thedriver in order to perform autonomous driving, based on the driveravailability and traffic hazard estimated at step S20.

When determining to take over the driving control from the driver (Yesat step S40), the control unit 100 may filter a driver input at stepS50, and move the vehicle to a safe area through autonomous driving atstep S60. That is, the control unit 100 may block the driver input suchthat the vehicle is controlled only through autonomous driving,establish a driving plan, and perform driving control to move thevehicle to a safe area. The driving plan may include checking a closesafe area, setting a path to the safe area, planning to change a line,performing road exit control, performing collision avoidance control,and performing emergency driving, and the driving control may includelane following, lane changing, pulling off the road and holding.

Then, the control unit 100 may hold the vehicle, and call for a help atstep S70. When the vehicle is positioned in the safe area after pullingoff the road, the control unit 100 may hold the vehicle, and call for ahelp through the communication unit 130. For example, a messageindicating the emergency situation may be autonomously transmitted to911.

On the other hand, when determining to maintain the driving control ofthe driver (No at step S40), the control unit 100 may return to step S10to continuously monitor the driver and the external environment.

In accordance with the embodiments of the present invention, the vehiclesafety support apparatus and method may determine a driving control bymonitoring a driver and an external environment, and move a vehicle to asafe area through autonomous driving when determining to take over thedriving control from the driver, thereby preventing a vehicle accident.

Hereafter, embodiments which can be applied when the driver availabilityestimation, the traffic hazard estimation, the driving controldetermination and the autonomous driving are performed will bedescribed. All, part or combinations of components described in thefollowing embodiments may be utilized for implementing the control unit100, the driver monitoring unit 110, the external environment monitoringunit 120, the communication unit 130, the GPS 140, the driver inputfiltering unit 150, the driver availability estimation unit 101, thetraffic hazard estimation unit 102, the driving control determinationunit 103 and the autonomous driving control unit 104, or implementingthe control logic or algorithm included in each of the units. In thefollowing embodiments, a control operation of changing a driving controlfor a vehicle from a driver to the control unit 100 may be performedwith an operation of warning the driver of a dangerous situation orcarelessness, and the vehicle safety support apparatus and method may bedesigned to determine the driving control depending on the level of thewarning.

The vehicle safety system (vehicle safety support apparatus) can bedivided into an environment recognition system and a vehicle controlsystem. The environment recognition system accurately recognizes theenvironment of vehicle and provides the recognized environmentinformation to the vehicle control system. The vehicle control systemsafely controls the vehicle using the environment information.

The environment recognition system should be able to handle varietyinformation which occurs during driving. Therefore, the environmentrecognition system requires high computational complexity and manysubsystems. Research is underway to efficiently control thesecalculations.

Hereinafter, exemplary embodiments will be described in detail withreference to the accompanying drawings.

FIG. 6 is a block diagram illustrating a system 1100 for managing adangerous driving index for vehicles, according to an embodiment of thepresent invention.

Referring to FIG. 6, the system 1100 for managing the dangerous drivingindex may not analyze a driving habit of a driver from a previouslylearned past driving pattern but may accurately analyze a driving habit,based on information which is acquired through various sensors in realtime.

The system 1100 for managing the dangerous driving index according to anembodiment of the present invention may use the information which isacquired through the various sensors in real time, and thus moreaccurately analyzes a driving habit of a driver than a related artdriving habit analysis method which analyzes a driving habit from alearned past driving pattern.

The system 1100 for managing the dangerous driving index may include aninternal sensor 1110, an external sensor 1120, an ambient environmentrecognizer 1130, a driving situation sensing interface 1140, a driverstatus sensing system 1160, and an output unit 1150.

The internal sensor 1110 may sense a driving situation of a driver'svehicle to acquire driver's vehicle driving information. The driver'svehicle driving information may include vehicle velocity information11-1, yaw rate information 11-3, steering angle information 11-5,acceleration information 11-7, and wheel velocity information 11-9.

In order to acquire the driver's vehicle driving information, theinternal sensor 1110 may include a vehicle velocity sensor 1110-1 thatacquires the vehicle velocity information 11-1, a yaw rate sensor 1110-3that acquires the yaw rate information 11-3, a steering angle sensor1110-5 that acquires the steering angle information 11-5, and a wheelvelocity sensor 1110-9 that acquires the wheel velocity information11-9.

The external sensor 1120 may sense an ambient situation of the driver'svehicle to acquire ambient environment information. The ambientenvironment information may include front/rear radar information 12-1,front/rear image information 12-3, side ultrasound information 12-5,around view monitoring (AVM) image information 12-7, and navigationinformation 12-9.

In order to acquire the ambient environment information, the externalsensor 1120 may include a front/rear radar 1120-1 that acquires thefront/rear radar information 12-1, a front/rear camera 1120-3 thatacquires the front/rear image information 12-3, a side ultrasoundgenerator 1120-5 that acquires the side ultrasound information 12-5, anAVM camera 1120-7 that acquires the AVM image information 12-7, anavigation (NAV) 1120-9 that acquires the navigation information 12-9, avehicle-to-infrastructure communication system and a vehicle-to-vehiclecommunication system. A vehicle-to-infrastructure communication systemand a vehicle-to-vehicle communication system may include information,for example, about traffic control feature (e.g., stop lights, stopsigns), current weather related information, information relating totransient anomalies transient anomalies and transient anomaly locations(e.g., construction zones, temporary speed limits, incident scenes(e.g., accident scenes, roadblocks, and so on), other vehicle'slocation, other vehicle's route (e.g., starting point, destination,expected trajectory), lane status, and cut in sign(e.g., turn signal).The lane status (information) can be one-lane county road, dividedhighway, boulevard, multi-lane road, one-way road, two-way road, or citystreet. Any variations of the above teachings are also intended to becovered by this patent application.

The ambient environment recognizer 1130 may calculate a trajectory loadby using the driver's vehicle driving information supplied from theinternal sensor 1110 and the ambient environment information suppliedfrom the external sensor 1120, and may manage a dangerous driving index,based on a result of the calculation. This will be described below indetail with reference to FIG. 7.

The driving situation sensing interface 1140 may interface the outputunit 1150 and provide a driver with driver status information suppliedfrom the driver status sensing system 1160 and the dangerous drivingindex through the output unit 1150.

The output unit 1150 may output the dangerous driving index, reflectedin the driving status information, in a visual or acoustical informationform to provide the dangerous driving index to the driver. To this end,the output unit 1150 may include a speaker 1150-1, an audio videonavigation (AVN) 1150-3, and a head up display 1150-5.

Moreover, the output unit 1150 may further include an engine controlsystem 1150-7, an automatic control system 1150-9, and a steeringcontrol system 1150-11, for adjusting a timing when the lateral controlof the vehicle starts.

The driver status sensing system 1160 may sense a driving status such asdrowsy driving, etc. The driver status sensing system 1160 will bedescribed below in detail with reference to FIGS. 15 to 22.

FIG. 7 is a block diagram illustrating a detailed configuration of theambient environment recognizer 1130 illustrated in FIG. 6.

Referring to FIG. 7, as described above, the ambient environmentrecognizer 1130 may calculate the trajectory load by using the driver'svehicle driving information supplied from the internal sensor 1110 andthe ambient environment information supplied from the external sensor1120, and calculate the dangerous driving index (or a peripheral riskindex), based on the calculated trajectory load.

In order to more accurately manage the dangerous driving index, theambient environment recognizer 1130 may calculate the dangerous drivingindex in further consideration of a peripheral vehicle load and a roadload in addition to the trajectory load.

The ambient environment recognizer 1130 may include a driver's vehicledriving trajectory generator 1130-1, a peripheral vehicle trajectorygenerator 1130-3, a trajectory load calculator 1130-5, a peripheralvehicle load calculator 1130-7, a road load calculator 130-9, and adangerous driving index manager 1130-11.

The driver's vehicle driving trajectory generator 1130-1 may acquire adriver's vehicle driving trajectory 1130-1 by using vehicle velocityinformation, steering angle information, reduction/accelerationinformation, and yaw rate information supplied from the internal sensor1110.

The peripheral vehicle trajectory generator 1130-3 may acquire aperipheral vehicle driving trajectory 13-3 by using the ambientenvironment information which includes the front/rear radar information12-1, the front/rear image information 12-3, the side ultrasoundinformation 12-5, and the AVM image information 12-7 supplied from theexternal sensor 1120.

The front/rear radar information 12-1 is low in accuracy of determiningan object, but enables accurate distance information (a longitudinaldirection) to be obtained. On the other hand, since the imageinformation 12-3 and 12-7 are used to acquire a monocular image, theimage information 12-3 and 12-7 are low in accuracy of the distanceinformation (the longitudinal direction), but the image information 12-3and 12-7 enable an object to be accurately determined and enable lateralinformation to be obtained.

In a target vehicle model equation, the longitudinal distanceinformation may be acquired by using the front/rear radar 1120-1, andthe lateral distance information may be acquired by using the front/rearcamera 1120-3, the AVM camera 1120-7, and the side ultrasound generator1120-5.

The following Equation (1) may be the target vehicle module equationwhich is used by the peripheral vehicle trajectory generator 130-3, forpredicting a peripheral vehicle trajectory.

$\begin{matrix}{{{A = \begin{bmatrix}1 & {\Delta\; t} & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & {\Delta\; t} \\0 & 0 & 0 & 1\end{bmatrix}},{H = \begin{bmatrix}1 & 0 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix}}}{x = \begin{bmatrix}x \\v_{x} \\y \\v_{y}\end{bmatrix}}{x_{k} = {{Ax}_{k} + w_{k}}}{z_{k} = {{Hx}_{k} + v_{k}}}} & (1)\end{matrix}$

where x, V_(x), y, and V_(y) denote status variables of a targetvehicle, and x and y denote a position of the target vehicle and aremeasured by an image camera. V_(x) and V_(y) denote a velocity of thetarget vehicle. A denotes a vehicle model equation, H denotes ameasurement value model equation, and the status variables respectivelydenote a distance and a velocity in an x axis direction and a distanceand a velocity in a y axis direction. A system noise and a measurementvalue noise denote white Gaussian.

The trajectory load calculator 1130-5 may calculate a trajectory load“W_(Trj)”. The trajectory load “W_(Trj)” may be a comparison resultwhich is obtained by comparing a predetermined threshold value with atrajectory distance value which is a difference between the peripheralvehicle trajectory 13-3 and the driver's vehicle driving trajectory13-1.

The driver's vehicle driving trajectory 13-1 and the peripheral vehicletrajectory 13-3 may be predicted, and a situation where there is a riskof collision may be a situation where a high caution of the driver isrequired. The trajectory load calculator 1130-5 may digitize thesituation as the trajectory load “W_(Trj)”.

The trajectory load “W_(Trj)” may be calculated as expressed in thefollowing Equation (2):W _(Trj)(i)=|T _(Trj)(i)−D _(Trj)(i)| if W _(Trj)(i)>Threshold,0W _(Trj)(i)<Threshold,1  (2)

where D_(Trj) denotes a driver's vehicle driving trajectory, and T_(Trj)denotes a peripheral vehicle trajectory. Also, i (1, 2, . . . , n)denotes a detected peripheral vehicle.

According to Equation (1), trajectories of detected peripheral vehiclesmay be compared with a trajectory of the driver's vehicle, and when atrajectory distance is less than a threshold value, the trajectory load“W_(Trj)” may be set to 1. Also, when the trajectory distance is greaterthan the threshold value, the trajectory load “W_(Trj)” may be set to 0.

The peripheral vehicle load calculator 1130-7 may analyze the number offront/rear/side peripheral vehicles and whether the peripheral vehicleschange lanes, based on the ambient environment information and maycalculate a peripheral vehicle load “W_(S)”, based on a result of theanalysis. The number of the peripheral vehicles and a trajectory changeof each of the peripheral vehicles may act as a load requiring a cautionof the driver.

In order to calculate the peripheral vehicle load, three dangeroussections {circle around (1)}, {circle around (2)} and {circle around(3)} may be calculated based on a time to collision (TTC). The threedangerous sections {circle around (1)}, {circle around (2)} and {circlearound (3)} are illustrated in FIG. 8. Here, the TTC may be defined as atime which is taken until a corresponding vehicle collides with a targetvehicle when a closing velocity of a vehicle is constant. The TTC may becalculated, based on the vehicle velocity information 11-1 and thesteering angle information 11-5.

For example, as illustrated in FIGS. 9 and 10, the three dangeroussections {circle around (1)}, {circle around (2)} and {circle around(3)} may be set by calculating a time which is obtained by a relativevelocity value of a detected vehicle by a TTC value for each ofperipheral vehicles (20, 30, 40 and 50 in FIG. 10) which are detectedfrom detection sections 43 and 47 detected by the front/rear radar1120-1, detection sections detected by the front/rear camera 1120-3, anda detection section 45 detected by the side ultrasound generator 1120-5.

When the three dangerous sections {circle around (1)}, {circle around(2)} and {circle around (3)} are set, the peripheral vehicle loadcalculator 1130-7 may analyze the number of peripheral vehicles detectedfrom the sections and whether the peripheral vehicles change lanes, andcalculate the peripheral vehicle load “W_(S)”, based on a result of theanalysis.

As the number of vehicles detected from the section {circle around (1)}increases and as the number of times the detected vehicles change lanesincreases, the peripheral vehicle load “W_(S)” may increase. On theother hand, when there is no detected peripheral vehicle, or althoughthere is a detected peripheral vehicle, the peripheral vehicle isdetected from the section {circle around (3)} or a trajectory change ofthe peripheral vehicle detected from the section {circle around (3)} isnot severe, the peripheral vehicle load “W_(S)” may decrease.

The peripheral vehicle load “W_(S)” may be expressed as the followingEquation (3):

$\begin{matrix}{W_{S} = {{\alpha{\sum\limits_{i = 1}^{n}S_{i}}} + {\beta{\sum\limits_{i = 1}^{n}L_{i}}}}} & (3)\end{matrix}$

where α denotes a weighting factor, β denotes a weighting factor, Sdenotes a position (the section {circle around (1)}, {circle around (2)}or {circle around (3)}) of a detected peripheral vehicle, and L denoteswhether the detected peripheral vehicle changes a lane. When thedetected peripheral vehicle has changed a lane, L may be set to 1, andwhen a lane is not changed, L may be set to 0. i (1<i<n, where n is anatural number) denotes a detected peripheral vehicle.

The road load calculator 1130-9 may calculate a road load by using aroad shape, a road surface status, and a traffic status which areincluded in the ambient environment information.

A caution of a driver is more required in a curve road than a straightroad and in a crossroad than a general road, and as a front trafficsituation becomes worse, a caution of the driver is required. Therefore,it is required to calculate the road load.

The road load may be calculated based on the navigation information12-9, which is supplied from the navigation 1120-9 and includes roadstatus information, and road surface status information acquired fromthe front/rear camera 1120-3. The road load may be calculated asexpressed in the following Equation (4):W _(R) =α×A+β×B+γ×C  (4)

where A denotes a value representing a road status. For example, as acurvature value of a front road increases, A may have a large value, andwhen traffic lights are changed, there is a pedestrian, a velocity islimited, or a current zone is a children protection zone, A may have alarge value. B denotes a road surface status value, and a paved road andan unpaved road may be reflected in B. C denotes traffic of a frontroad, and as traffic increases, C may have a large value. A, B, and Cmay be all normalized to a range of 0 to 5.

The dangerous driving index manager 1130-11 may manage the dangerousdriving index (the peripheral risk index) obtained by summating theplurality of loads “W_(Trj)”, “W_(S)” and “W_(R)” which are calculatedin respective steps.

The dangerous driving index manager 1130-11 may summate the trajectoryload “W_(Trj)”, the peripheral vehicle load “W_(S)”, and the road load“W_(R)” to calculate a summating result value as the dangerous drivingindex.

When the calculated dangerous driving index is higher than apredetermined threshold value, the dangerous driving index manager1130-11 may supply the calculated dangerous driving index to the vehiclecontrol systems 1150-7, 1150-9 and 1150-11 through the driving situationsensing interface 1140.

The vehicle control systems 1150-7, 1150-9 and 1150-11, which havereceived the calculated dangerous driving index higher than thepredetermined threshold value, may control an engine operation, abraking operation, and a steering operation to restrict a drivingfunction of the vehicle.

The dangerous driving index may be expressed as the following Equation(5):Dangerous Driving Index=W _(Trj) +W _(S) +W _(R)  (5)

FIG. 11 is a flowchart illustrating a method of managing a dangerousdriving index for vehicles, according to an embodiment of the presentinvention.

Referring to FIG. 11, in step 1110, an operation of generating adriver's vehicle driving trajectory may be performed, and in step S1112,an operation of predicting a position of the driver's vehicle by usingthe generated driver's vehicle driving trajectory may be performed. Thedriver's vehicle driving trajectory may be generated based on vehiclevelocity information, steering angle information, reduction/accelerationinformation, and yaw rate information.

Subsequently, in step S1114, an operation of generating a peripheralvehicle driving trajectory may be performed, and in step S1116, anoperation of predicting a position of a peripheral vehicle may beperformed. The peripheral vehicle driving trajectory may be generatedbased on longitudinal distance information acquired from a radar,lateral distance information acquired from a camera, and lateraldistance information acquired from an ultrasound generator. Here, thelongitudinal distance information may be longitudinal distanceinformation to the peripheral vehicle with respect to the driver'svehicle, and the longitudinal distance information may be longitudinaldistance information to the peripheral vehicle with respect to thedriver's vehicle.

Subsequently, in step S1118, an operation of calculating a trajectoryload may be performed. The trajectory load may be calculated based onthe driver's vehicle driving trajectory and the peripheral vehicledriving trajectory. For example, driving trajectories of detectedperipheral vehicles may be compared with the driver's vehicle drivingtrajectory, and when a trajectory distance which is a differencetherebetween is less than a threshold value, the trajectory load may becalculated as 1. On the other hand, when the trajectory distance isgreater than the threshold value, the trajectory load may be calculatedas 0.

Subsequently, in step S1120, an operation of calculating a peripheralvehicle load may be performed. The peripheral vehicle load may becalculated in consideration of the number of vehicles, located in eachof a plurality of dangerous sections which are divided based on a TTC,and whether the vehicles change lanes. The peripheral vehicles locatedin the plurality of dangerous sections may be detected by using a radar,a camera, and an ultrasound generator, and the plurality of dangeroussections may be obtained by calculating a time which is obtained bydividing a relative velocity value by a relative distance (which is aTTC value) to a detected vehicle.

Subsequently, in step S1122, an operation of calculating a road load maybe performed. The road load may be calculated based on navigationinformation, road surface status information, traffic information, etc.

Subsequently, in step S1124, an operation of calculating a dangerousdriving index may be performed. The dangerous driving index may becalculated by summating the trajectory load, the peripheral vehicleload, and the road load.

Subsequently, in step S1126, an operation of comparing the calculateddangerous driving index with a threshold value may be performed. Whenthe dangerous driving index is equal to or greater than the thresholdvalue, the system 1100 may warn the driver of a dangerous drivingsituation by steps, based on a level of the threshold value in stepS1128. Here, when the dangerous driving index is less than the thresholdvalue, steps S1110 to S1124 may be again performed.

In step S1130, an operation of providing the dangerous driving indexindicating the dangerous driving situation to the driver through aspeaker, a HUD, and an AVN equipped in the vehicle may be performed.

FIG. 12 is a block diagram for describing another embodiment of theambient environment recognizer illustrated in FIG. 7.

Referring to FIG. 12, an ambient environment recognizer 1130 accordingto another embodiment of the present invention may include a detectionsection generator 1132 for detecting an optimized section region(hereinafter referred to as a detection section) and recognizing anambient environment in the optimized detection section. Other elementsexcept the detection section generator 1132 are the same as the elementsincluded in the ambient environment recognizer 1130 of FIG. 7, and thus,the details of the ambient environment recognizer 1130 described abovewith reference to FIG. 7 may be applied to the other elements.

The detection section generator 1132 may optimize a detection sectionincluding a target object so as to accurately distinguish an actualobstacle among target objects around a vehicle.

FIG. 13 is a block diagram illustrating a detailed configuration of thedetection section generator 1132 illustrated in FIG. 12.

As illustrated in FIG. 13, the detection section generator 1132 foroptimizing a detection section according to an embodiment of the presentinvention may include a remote sensor 1132-1, a target tracer 1132-3, atrack manager 1132-5, and a storage 1132-9.

The remote sensor 1132-1 may detect a position of an object around avehicle to output a detection signal. In this case, the remote sensor1132-1 may include one or more of a lidar sensor, a radar sensor, and acamera sensor.

The target tracer 1132-3 may distinguish an obstacle, based on thedetection signal and generate a track which includes a covariance of anerror and a position estimation value corresponding to the distinguishedobstacle, thereby tracing the position.

Generally, a Kalman filter may be used for overcoming an error of asensor and sensing a position of a moving object.

The Kalman filter may use a technique that repeats an operation ofcalculating an estimation value of a positon of an object, based on anestimation value of a position of the object at a previous time and ameasurement value of a position of the object and thus counteracts anerror which occurs in measuring the position of the object, therebyestimating an accurate position of the object. In this case, anestimation value at a current time which is based on only a measurementvalue to a previous time may be calculated based on an estimation valueof a position of an object to the previous time.

Subsequently, an estimation value at a current time which is based ononly a measurement value to a previous time may be corrected based on acovariance at the current time, which is calculated based on only themeasurement value to the previous time, and a measurement value of aposition of an object at the current time, and an estimation value ofthe position of the object at the current time may be calculated.

The target tracer 1132-3 may set the number of tracks and a detectionsection and an initial position of an obstacle corresponding to each ofthe tracks, based on the position of the object indicated by thedetection signal. In this case, the detection section and the initialposition of the obstacle may be calculated by the Kalman filter asexpressed in the following Equation (6):{circumflex over (x)}(k|k−1)=F(k−1){circumflex over (x)}(k−1|k−1){circumflex over (z)}(k|k−1)=H(k){circumflex over (x)}(k|k−1)  (6)

where {circumflex over (x)}(k|k−1) denotes an estimation value of astatus value of the object at a time “k” which is estimated based oninformation to a time “k−1”, {circumflex over (x)}(k−1|k−1) denotes anestimation value of a status value of the object at the time “k−1” whichis estimated based on the information to the time “k−1”, and {circumflexover (z)}(k|k−1) denotes an estimation value of a position of the objectat the time “k” which is estimated based on the information to the time“k−1”.

Here, each track may include the Kalman filer for tracing a specificobject sensed by the remote sensor 1132-1. That is, each track mayinclude a covariance of an error for correcting a position on the basisof a measurement value and an estimation value of a position of a tracedobstacle, and an estimation value, a measurement value, and a covariancewhich are calculated per time may be stored as histories in the storage1132-9.

A configuration of the Kalman filter included in each track may beexpressed as the following Equation (7):

$\begin{matrix}{\mspace{79mu}{{{{x(k)} = {{{F\left( {k - 1} \right)}{x\left( {k - 1} \right)}} + {v\left( {k - 1} \right)}}}\mspace{79mu}{{z(k)} = {{{H(k)}{x(k)}} + {w(k)}}}{{P\left( k \middle| k \right)} = {\sum\limits_{i}{{\beta\left( {k,i} \right)}\left\lbrack {{P\left( {\left. k \middle| k \right.,i} \right)} + {\left( {{\hat{x}\left( {\left. k \middle| k \right.,i} \right)} - {\hat{x}\left( k \middle| k \right)}} \right)\left( {{\hat{x}\left( {\left. k \middle| k \right.,i} \right)} - {\hat{x}\left( k \middle| k \right)}} \right)^{T}}} \right\rbrack}}}}\mspace{20mu}{{P\left( k \middle| {k - 1} \right)} = {{{F\left( {k - 1} \right)}{P\left( {k - 1} \middle| {k\; 1} \right)}{F\left( {k - 1} \right)}^{T}} + {Q\left( {k - 1} \right)}}}}} & (7)\end{matrix}$

where x(k) denotes a status value of the object at the time “k”, F(k−1)denotes a status change model representing a change when a time ischanged from the time “k−1” to the time “k”, z(k) denotes a position ofthe object at the time “k”, H(k) denotes an observation modelrepresenting a change from a statue of the object to the position of theobject, v(k−1) denotes a processing noise at the time “k−1”, and w(k)denotes a measurement noise at the time “k”. Also, P(k|k) denotes acovariance of an error of the Kalman filter at the time “k” which iscalculated based on information to the time “k”, and P(k|k−1) denotes acovariance of an error of the Kalman filter at the time “k” which iscalculated based on information to the time “k−1”. Q(k−1) denotes aprediction covariance at the time “k−1”.

The target tracer 1132-3 may determine whether the position of theobject indicated by the detection signal is included in the detectionsection corresponding to a track, based on errors of a measurement valueand an estimation value of the position of the object and a covarianceof each of the errors.

In this case, the target tracer 1132-3 may set a range of the detectionsection, based on a status value of the Kalman filter included in acurrent track and update a status value of the Kalman filter, based onmeasurement values included in the detection section. The target tracer1132-3 may calculate a residual based on the measurement value and theestimation value of the position of the object, calculate a residualcovariance, based on the observation model and a covariance of anestimation error included in the Kalman filter and determine whether theobject enters the detection section, based on the residual and theresidual covariance.

Here, the detection section may be set as a section which represents aspecific probability value or less in the Gaussian probabilitydistribution having the residual covariance as a variance, and theprobability value may be referred to as a gate probability. Therefore,the detection section may be calculated by calculating the residualcovariance and setting a gate probability value, and the residualcovariance and the gate probability value may be optimized by the Kalmanfilter with time, whereby the detection section may be optimized withtime.

A method of calculating the residual and the residual covariance and acondition where the object is located in the detection section may beexpressed as the following Equation (8):v(k,i)=z(k,i)−{circumflex over (z)}(k|k−1)S(k)=H(k)P(k|k−1)H(k)^(T) +R(k)v(k,i)^(T) S(k)⁻¹ v(k,i)<r  (8)

where v(k, i) denotes a residual of an object “i” at a time “k”, andz(k, i) denotes a measurement value of a position of the object “i”.Also, P(k|k−1) denotes a covariance of an estimation error of the Kalmanfilter, R(k) denotes a measurement noise at the time “k”, S(k) denotesan estimation noise at the time “k”, and r denotes a range of adetection section.

The track manager 1132-5 may update a position estimation value includedin a track, based on a detection signal included in the trackcorresponding to an object which is located in a detection section.

In this case, in order to update the position estimation value, thetrack manager 1132-5 may calculate a Kalman gain, based on a residualcovariance and a covariance of an estimation error and calculate theposition estimation value by using information to a current is time,based on an estimation value of a position which is estimated based onthe Kalman gain, a position measurement value of an object, andinformation to a previous time. The update of the position estimationvalue may be expressed as the following Equation (9):

$\begin{matrix}{{{K(k)} = {{P\left( k \middle| {k - 1} \right)}H^{T}{S(k)}^{- 1}}}{{\hat{x}\left( {\left. k \middle| k \right.,i} \right)} = \left\{ {{\begin{matrix}{\hat{x}\left( k \middle| {k - 1} \right)} & {i = 0} \\{{\hat{x}\left( k \middle| {k - 1} \right)} + {{K(k)}{v\left( {k,i} \right)}}} & {i > 0}\end{matrix}{\hat{x}\left( k \middle| k \right)}} = {\sum\limits_{i}{{\beta\left( {k,i} \right)}{\hat{x}\left( {\left. k \middle| k \right.,i} \right)}}}} \right.}} & (9)\end{matrix}$

where K(k) denotes the Kalman gain.

As described above, the track manager 1132-5 may update the positionmeasurement value with time, based on the measurement value, therebycalculating a more accurate position estimation value.

When a distance between an object position estimation value included ina first track and an object position estimation value included in asecond track is less than a predetermined reference value, the trackmanager 1132-5 may initialize the first track and the second track,based on a history stored in the storage 1132-9.

The storage 1132-9 may store a history which is obtained by updating atrack. In this case, the history stored in the storage 1132-9 mayinclude a position estimation value, a position measurement value, and acovariance value of an estimation error with time of the Kalman filterincluded in the track.

When a position estimation value is updated as described above, objectsindicated by two tracks may collide with each other depending on thecase. When a position estimation value indicated by an object is reducedto less than a pre-stored reference value, the track manager 1132-5 maydetermine that the objects indicated by two tracks collide with eachother, and initialize the track, based on data included in histories ofthe two colliding tracks.

Moreover, when all object position estimation values included in a trackare not included in a detection section corresponding to the track, thetrack manager 1132-5 may initialize the track, based on a history of thetrack stored in the storage 1132-9. That is, when an object traced by atrack deviates from the detection section, or the object traced by thetrack is determined as noise or an error and thus disappears, the trackmay fail to trace the object, and thus, the track manager 132-5 mayinitialize the track and trace a new object.

As described above, a track may trace a moving obstacle by using theKalman filter. When a track tracing an object fails to trace the objector two tracks collide with each other, a track may be initialized, and anew object may be traced. Accordingly, an object identificationperformance of a peripheral status detection system is enhanced.

As described above, the target tracer 1132-3 and the track manager1132-5 may trace an obstacle to generate or update a track, and dataincluded in the generated or updated track may be transferred to thevehicle controller 1132-7 and may be used to control a vehicle in orderfor the vehicle to avoid an obstacle or issue a warning.

FIG. 14 is a flowchart illustrating a method of optimizing a detectionsection, according to an embodiment of the present invention.

Referring to FIG. 14, in step S1410, the target tracer 1132-3 maygenerate a track which includes a covariance of an error and a positionestimation value corresponding to a distinguished obstacle, based on adetection signal which the remote sensor 1132-1 senses a position of anobject and outputs.

In this case, as described above, the remote sensor 1132-1 may includeone or more of a lidar sensor and a radar sensor.

Moreover, a track generated by the target tracer 1132-3 may include aKalman filter which includes a position estimation value and acovariance of an error. In this case, a configuration of the Kalmanfilter included in the track is as described with reference to Equations(1) and (2).

Subsequently, in step S1420, the target tracer 1132-3 may calculate adetection section which is a range where an obstacle is detected for thetrack.

In this case, a size of the detection section may be set to an initialvalue, based on a position of the object indicated by the detectionsignal. Also, the detection section may be set as a section whichrepresents a gate probability value or less in the Gaussian probabilitydistribution having a residual covariance as a variance.

Subsequently, in step S1430, the target tracer 1132-3 may select a validdetection signal, where the position of the object indicated by thedetection signal is included in the detection section, from thedetection signal.

As described above, the detection signal may include a measurement valueof the position of the object traced by a peripheral search system of avehicle, and the target tracer 1132-3 may select a valid measurementvalue of the measurement value included detection section to update theKalman filter, and use the updated Kalman filter for tracing an object.

In this case, the target tracer 1132-3 may determine whether theposition of the object indicated by the detection signal is included inthe detection section corresponding to the track, based on errors of ameasurement value and an estimation value of the position of the objectand a covariance of each of the errors.

The target tracer 1132-3 may set a range of the detection section, basedon a status value of a Kalman filter included in a current track andupdate a status value of the Kalman filter by using measurement valuesincluded in the detection section. Here, the target tracer 1132-3 maycalculate a residual based on the measurement value and the estimationvalue of the position of the object, calculate a residual covariance,based on the observation model and a covariance of an estimation errorincluded in the Kalman filter and determine whether the object entersthe detection section, based on the residual and the residualcovariance. A method of calculating the residual and the residualcovariance and a condition where the object is located in the detectionsection are as expressed in Equation (8). The residual covariance and agate probability value may be optimized by the Kalman filter with time,and thus, the detection section may be optimized with time.

Subsequently, in step S1440, when a distance between an object positionestimation value included in a first track and an object positionestimation value included in a second track is less than a predeterminedreference value, the track manager 1132-5 may initialize the first trackand the second track, based on a history stored in the storage 1132-9.

The history stored in the storage 1132-9 may include a positionestimation value, a position measurement value, and a covariance valueof an estimation error with time of the Kalman filter included in thetrack.

When a position estimation value is updated as described above, objectsindicated by two tracks may collide with each other depending on thecase. When a position estimation value indicated by an object is reducedto less than a pre-stored reference value, the track manager 1132-5 maydetermine that the objects indicated by two tracks collide with eachother, and initialize the track, based on data included in histories ofthe two colliding tracks.

Subsequently, in step S1450, the track manager 1132-5 may update theselected detection signal and a position estimation value included in atrack corresponding to an object of which a position is located in thedetection section.

In this case, in order to update the position estimation value, thetrack manager 1132-5 may calculate a Kalman gain, based on a residualcovariance and a covariance of an estimation error and calculate theposition estimation value by using information to a current time, basedon an estimation value of a position which is estimated based on theKalman gain, a position measurement value of an object, and informationto a previous time. The update of the position estimation value is asexpressed in Equation (9).

Subsequently, when all object position estimation values included in atrack are not included in a detection section corresponding to thetrack, the track manager 1132-5 may initialize the track, based on ahistory of the track stored in the storage 1132-9 and terminate aprocess.

That is, when an object traced by a track deviates from the detectionsection, or the object traced by the track is determined as noise or anerror and thus disappears, the track may fail to trace the object, andthus, the track manager 1132-5 may initialize the track and trace a newobject.

As described above, a track may trace a moving obstacle by using theKalman filter. When a track tracing an object fails to trace the objector two tracks collide with each other, a track may be initialized, and anew object may be traced. Accordingly, an object identificationperformance of a peripheral status detection system is enhanced.

Data included in a track which is generated or updated by theabove-described method may be transferred to the vehicle controller1132-7 and may be used to control a vehicle in order for the vehicle toavoid an obstacle or issue a warning.

As described above, a vehicle control apparatus and method according toan embodiment of the present invention dynamically updates a valid gaterepresenting a section of interest (SOI) which is traced by theperipheral status detection system of a vehicle for sensing an obstacle,and thus accurately traces an obstacle around the vehicle. Therefore,the vehicle control apparatus and method extend a distance to anobstacle of which a position is accurately traced by using only a lidarsensor or a radar sensor, thereby preventing an accident.

FIG. 15 is a block diagram illustrating a detailed configuration of thedriver status sensing system 1160 illustrated in FIG. 1.

Referring to FIG. 15, the driver status sensing system 1160 may includean acquisition unit 1161, a control unit 1163, and an output unit 1165.

The acquisition unit 1161 may be an element for acquiring drivingmanipulation information of a vehicle and careless status information ofa driver. In this case, the acquisition unit 1161 may include a drivingmanipulation sensing unit 1161A and a careless status sensing unit1161B. The acquisition unit 1161 may acquire the driving manipulationinformation by using the driving manipulation sensing unit 1161A andacquire the careless status information by using the careless statussensing unit 1161B.

The driving manipulation sensing unit 1161A may be an element forsensing manipulation of a control unit which necessarily operates fordriving a vehicle. For example, the driving manipulation sensing unit1161A may be an electronic control unit (ECU) of the vehicle or aseparate module which is included in the ECU.

The driving manipulation sensing unit 1161A may include a plurality ofmanipulation sensing units such as an acceleration pedal manipulationsensing unit d₁, a brake pedal manipulation sensing unit d₂, amulti-function manipulation sensing unit d₃, and a steering handlemanipulation sensing unit d₄.

As illustrated in (a) FIG. 16, the plurality of manipulation sensingunits included in the driving manipulation sensing unit 1161A may sensemanipulation for driving of the vehicle, and for example, may sense atleast one of acceleration pedal (ACC pad) manipulation, brake pedalmanipulation, multi-function switch manipulation, and steering handlemanipulation.

In addition, the driving manipulation sensing unit 1161A may acquire thedriving manipulation information which further includes clutch pedalmanipulation or transmission manipulation of a manual-gear vehicle. Inthis case, the driving manipulation sensing unit 1161A may check avelocity of the vehicle, and when the velocity of the vehicle is equalto or higher than a certain velocity (for example, 10 km/h), the drivingmanipulation sensing unit 1161A may acquire the driving manipulationinformation.

For example, the driving manipulation sensing unit 1161A may sense thenumber “n_(A)” of operations of an acceleration pedal manipulated by adriver at every certain time for a predetermined time. For example, thedriving manipulation sensing unit 1161A may continuously check whetherthe acceleration pedal operates (ON), at every second of 50 ms forsecond of 200 ms.

Likewise, the driving manipulation sensing unit 1161A may sense thenumbers “n_(B)” and “n_(C)” of operations of the brake pedal and theclutch pedal which are manipulated at every certain time.

As another example, the driving manipulation sensing unit 1161A maysense the number “n_(M)” of operations of each multi-function switchwhich is manipulated by a driver for a certain time. Here, eachmulti-function switch may be a switch for operating a vehicle wiper or avehicle lamp such as a turn signal lamp. For example, the drivingmanipulation sensing unit 1161A may count the number of operations of amulti-function switch which is manipulated for second of 200 ms.

As another example, the driving manipulation sensing unit 1161A maysense an angular velocity of a steering wheel which is manipulated bythe driver for a certain time. In this case, the driving manipulationsensing unit 1161A may measure an angle change amount of the steeringwheel to calculate the angular velocity without separately measuring theangular velocity. For example, the driving manipulation sensing unit1161A may measure the angle change amount of the steering wheel tocalculate the angular velocity at every second of 50 ms for second of200 ms.

The careless status sensing unit 1161B may be an element for sensing amotion of the driver and manipulation of a control unit whichadditionally operates when the vehicle drives. The careless statussensing unit 1161B may include a plurality of sensing units such as anaudio signal input sensing unit T₁, an air conditioning signal inputsensing unit T₂, a navigation signal input sensing unit T₃, etc.

Moreover, the careless status sensing unit 1161B may include adrowsiness index measuring unit P, a viewing direction measuring unit E,and a voice sensing unit V, for sensing a motion (a visual factor or averbal factor) of the driver. The plurality of sensing units and theplurality of measuring units included in the careless status sensingunit 1161B, as illustrated in FIG. 15, may be provided at a certainposition of the vehicle and may acquire careless status information,based on information about at least one of manipulation of a peripheraldevice, a facial image of the driver, and a voice of the driver.

For example, the careless status sensing unit 1161B may sense the number“n_(T)” of manipulations of peripheral devices which are made by thedriver while the vehicle is driving. Here, the peripheral devices maynot be devices essential for driving of the driver unlike an AVN, avehicle air conditioning device, etc., but may be control units whichare manipulated for convenience of the driver or a vehicle indoorenvironment. For example, the careless status sensing unit 1161B maysense the number of inputs of operational switches of peripheraldevices, which are manipulated by the driver for a certain time whilethe vehicle is driving at a certain velocity (for example, 10 km/h) ormore, to acquire careless status information.

As another example, the careless status sensing unit 1161B may sense avoice of the driver through a microphone which is disposed at a certainposition in the vehicle. For example, the careless status sensing unit1161B may check a voice production time and a pulse level (a voicelevel) of voice data which is received from the microphone for a certaintime. For example, the careless status sensing unit 1161B may detect atime, at which voice data having a pulse level equal to or higher than apulse threshold value is received for a certain time (for example, 200ms), to acquire careless status information by using the pulse thresholdvalue stored in a memory.

As another example, the careless status sensing unit 1161B may receive afacial image of the driver from a camera, which is disposed at a certainposition of the vehicle, to acquire eye-closing information andobservation negligence information of the driver. In this case, asillustrated in (b) of FIG. 16, the camera may include a near infraredlight emitting diode (LED) for recording an image at daytime andnighttime.

For example, the careless status sensing unit 1161B may separatelyextract an eye region image of the driver from the facial image of thedriver. In this case, the eye region image may be extracted from thefacial image of the driver through image processing. That is, thecareless status sensing unit 1161B may acquire careless statusinformation such as the eye closing and observation negligence of thedriver, based on the facial image of the driver acquired from the cameraand the eye region image included in the facial image.

The careless status sensing unit 1161B may acquire the eye-closinginformation of the driver, based on the eye region image extracted fromthe facial image of the driver.

Referring to FIG. 17, the careless status sensing unit 1161B may sensean eyelid region from the eye region image, and when a sum of ∠A and ∠Bwhich are angles of eyelids is equal to or less than ∠C which is apredetermined threshold angle value (∠A+∠B≤∠C), the careless statussensing unit 1161B may determine the driver as closing eyes.

The careless status sensing unit 1161B may check the angles of theeyelids for a certain time to sense the number of times the drivercloses eyes, and calculate a certain time value and the number of timesthe driver closes the eyes, thereby acquiring a drowsing time (adrowsiness index) of the driver. For example, the careless statussensing unit 1161B may measure (count) the number of eye-closings bydividing one second into sections of 250 ms, and when a counting valueis 3, a time (the drowsiness index) when the driver closes the eyes maybe acquired as 750 ms.

Moreover, the careless status sensing unit 1161B may acquire theobservation negligence of the driver, based on the facial image of thedriver and the eye region image included in the facial image.

Referring to FIG. 20, in an observation negligence range, observationnegligence information may be acquired by checking (a viewing direction)whether a viewing range of the driver is within a visual distance “η”where there is no load when the vehicle drives, based on an angle of avehicle wheel (i.e., an angle change amount (ΘA or ΘB in (a) of FIG. 20)of a wheel in a center of the vehicle) instead of an angle of a steeringwheel. The viewing range may be checked by performing an operation whichcalculates an angle of a face (α in (b) of FIG. 20) from the facialimage of the driver acquired from the camera and then measures aposition (a position of pupil) of a pupil (β in (c) of FIG. 20) in theeye region image.

In detail, in a case where a gear lever is D gear or N gear when anangle of the steering wheel is within a certain angle, when the viewingof the driver does not enter a driving non-load visual range (a, b) fora certain time, the careless status sensing unit 1161B may determineobservation negligence, check a corresponding time, and acquire theobservation negligence information. For example, in a case where anangle of the steering wheel is less than ±15 degrees when a velocity ofthe vehicle is 10 Km or more and the gear lever is the D gear or the Ngear, when the viewing of the driver does not enter the driving non-loadvisual range (a, b) for 1.5 seconds or more, the careless status sensingunit 1161B may determine observation negligence for the front of thevehicle.

The control unit 1163 may be an element that controls an overalloperation of the driver status sensing system 1160, and may be anelectronic control unit. For example, the control unit 1163 may bereferred to as a driving workload compute unit (DWC).

In detail, the control unit 1163 may calculate a driving manipulationload and a driving interruption load, based on the driving manipulationinformation and the careless status information acquired from theacquisition unit 1161.

Moreover, the control unit 1163 may compare the driving manipulationload with the driving interruption load to determine whether a drivingstatus of the driver is a safety driving status, and when it isdetermined that the driving status of the driver is not the safetydriving status, the control unit 1163 may control the output unit 1165to output warning information.

First, the control unit 1163 may calculate the driving manipulation loadand the driving interruption load, based on the driving manipulationinformation and the careless status information acquired from theacquisition unit 1161.

For example, the control unit 1163 may calculate the drivingmanipulation load “W_(d)”, based on the driving manipulation informationacquired from the driving manipulation sensing unit 1161A of theacquisition unit 1161.

In detail, the control unit 1163 may acquire the driving manipulationinformation from the driving manipulation sensing unit 1161A, read anitem-based weight value of the driving manipulation information from thememory, and calculate the driving manipulation load.

Here, the weight value may be a value which is extracted andpredetermined for each item of driving manipulation information throughvarious experiments. Alternatively, the weight value may be a valuewhich is arbitrarily predetermined for each item by a worker. Also, thememory may be a storing means for storing data, and for example, may bea nonvolatile memory.

Moreover, the control unit 1163 may calculate a load for each item,based on an item included in the driving manipulation information and aweight value corresponding to the item. For example, when drivingmanipulation information of acceleration pedal manipulation, brake pedalmanipulation, multi-function switch manipulation, and steering handlemanipulation is acquired from the acquisition unit 1161, the controlunit 1163 may read, from the memory, a weight value “D_(A)” of theacceleration pedal manipulation, a weight value “D_(B)” of the brakepedal manipulation, a weight value “D_(M)” of the multi-function switchmanipulation, and a weight value “θ” of the steering handlemanipulation.

For example, when time information and times information of each itemare acquired from the acquisition unit 1161 in units of 50 ms for secondof 200 ms, the driving manipulation load “W_(d)” calculated by thecontrol unit 1163 for second of 200 ms may be expressed as the followingEquation (10):W _(d) =D _(A) ×n _(A)×50 ms+D _(B) ×n _(B)×50 ms+D _(C) ×n _(C)×50 ms+D_(M) ×n _(M) +θ×n _(θ)×50 ms  (10)

Each item included in the driving manipulation information may be addedor subtracted depending on the case. For example, when clutch pedalmanipulation and transmission manipulation are included in the drivingmanipulation information depending on the kind of a vehicle, the controlunit 1163 may calculate the driving manipulation load in furtherconsideration of a weight value of the clutch pedal manipulation and aweight value of the transmission manipulation.

As another example, the control unit 1163 may calculate the drivinginterruption load “W_(i)”, based on the careless status informationacquired from the careless status sensing unit 1161B of the acquisitionunit 1161. In detail, the control unit 1163 may calculate loads of theitems included in the careless status information acquired from thecareless status sensing unit 1161B, and summate the calculated loads ofthe items to calculate the driving interruption load.

The control unit 1163 may calculate a peripheral device manipulationload “T(n)”, based on the number of manipulations of peripheral devicesacquired from the careless status sensing unit 1161B of the acquisitionunit 1161.

For example, the control unit 1161 may calculate the peripheral devicemanipulation load in further consideration of a weight value ofperipheral device manipulation which is stored in the memory. Forexample, when the number of inputs of peripheral device manipulationwhich is made by the driver for second of 20 ms is acquired from theacquisition unit 1161, the control unit 1163 may perform an arithmeticoperation on the number of inputs of the peripheral device manipulationand a weight value to calculate the peripheral device manipulation load“T(n)”.

Furthermore, the control unit 1163 may calculate a voice load “V(n)”,based on voice data of the driver acquired from the careless statussensing unit 1161B of the acquisition unit 1161. In detail, the controlunit 1163 may calculate the voice load, based on a pulse threshold valueof the voice data stored in the memory. For example, the control unit1163 may calculate a time, at which voice data having the pulsethreshold value or more is received, in the voice data of the driverwhich is acquired from the careless status sensing unit 1161B of theacquisition unit 1161 for a certain time (for example, 200 ms), therebycalculating a voice load “V(t)”.

Moreover, the control unit 1163 may calculate a drowsiness load “P(t)”and a driving observation negligence load “E(t)”, based on theeye-closing information and the observation negligence information whichare included in the careless status information acquired from thecareless status sensing unit 1161B of the acquisition unit 1161.

Referring to a graph of FIGS. 18A to 18F, when the driver is in a tardystate, a change width of the graph may be very stably narrow, but aPERCLOS value when the driver is in a drowsy status may be very severein change and may be higher in whole numerical value than the tardystatus. Here, the PERCLOS value may be expressed as the followingEquation (11):

$\begin{matrix}{{{PERCLOS}(\%)} = {\frac{{{accumulation}\mspace{14mu}{of}\mspace{14mu}{eye}} - {{closing}\mspace{14mu}{time}}}{{certain}\mspace{14mu}{measurement}\mspace{14mu}{time}\mspace{14mu}{for}\mspace{14mu}{accumulation}} \times 100}} & (11)\end{matrix}$

Based on the graph of FIGS. 18A to 18F, when it is checked by thecontrol unit 1163 that the PERCLOS value is a certain percentage (30%)or more, namely, it is checked through the careless status sensing unit1161B that the driver have closed eyes for a certain time (for example,75 seconds) with respect to a certain time (for example, 250 seconds),the control unit 1163 may control the output unit 1165 to output avehicle warning sound. This is for immediately outputting a warningsound to a driver by determining the driving of the driver as drowsydriving when it is checked that the driver has closed eyes for a certaintime or more because the drowsy driving is the highest dangerous factorin driving a vehicle.

Hereinafter, an operation where the control unit 1163 checks the eyeclosing of the driver will be described with reference to a flowchart ofFIG. 19.

In detail, in step S1910, the control unit 1163 may determine whether avehicle velocity “V” is a certain velocity (for example, V≥10 Km/h) ormore. When it is determined that the vehicle velocity “V” is the certainvelocity or more, by using the careless status sensing unit 1120, thecontrol unit 1163 may check the number “y” of eye closings from an angleof an eyelid per certain time “x” (for example, 250 ms or 0.25 s) for acertain measurement time “N” to calculate an eye closing time (adrowsing time) of the driver.

In step S1920, the control unit 1163 may determine whether the angle ofthe eyelid is ∠A+∠B≤∠C. When the angle of the eyelid is ∠A+∠B≤∠C, thecontrol unit 1163 may determine the driver as closing the eyes to count(y++) the number of eye closings in step S1930.

The control unit 1163 may count (x+=0.25, y++) the number “y” of eyeclosings per certain time “x”, and when the number of eye closings isthree (x=1, y=3) for one second in step S1940, the control unit 1163 maycount (P++) a patient factor value “P” and repeat the operation whileincreasing (N++) a time by one second in step S1950.

For example, when the patient factor value “P” is 100 or more within ameasurement time of 250 seconds in step S1960, the control unit 1163 maycontrol the output unit 1165 to generate a warning event in step S1970.At this time, the control unit 1163 may change the patient factor value“P” to 99, reduce (P=99, N-=1) the measurement time “N” by one second,repeat the above-described steps, and calculate an eye closing time (adrowsing time) of the driver. In addition, the control unit 1163 maycheck a case where the PERCLOS value is 30% or more for a certain time,thereby calculating the drowsiness load “P(t)”.

Moreover, the control unit 1163 may calculate the observation negligenceload “E(t)”, based on the observation negligence information acquiredfrom the careless status sensing unit 1161B.

As illustrated in (a) of FIG. 20, when the viewing range deviates fromthe driving non-load visual distance “η”, a factor value of the loadfactor “P” may be changed according to a deviating range.

When a deviating range (a′, b′) is predetermined, the control unit 1163may read a factor value of a load factor from the memory for each rangecorresponding to a current viewing range of the driver, therebycalculating the observation negligence load “E(t)”.

The control unit 1163 may perform an arithmetic operation on thecalculated peripheral device manipulation load “T(n)”, voice load“V(t)”, drowsiness load “P(t)”, and observation negligence load “E(t)”to calculate the driving interruption load “W_(i)”.

Moreover, depending on a status of the vehicle and the case, at leastone of loads which are acquired based on the careless status informationmay not be considered for calculating the driving interruption load.

The control unit 1163 may compare the driving load “W_(d)” with thedriving interruption load “W_(i)” to determine a safety driving statusof the driver.

In detail, when a value of the driving interruption load “W_(i)”subtracted from the driving load “W_(d)” is equal to or less than apredetermined spare load “W_(l)”, the control unit 1163 may determinethat the driver is not in a safety driving status. Here, thepredetermined spare load “W_(l)” may be a value which is extracted froma driving load and driving information (including a change gear status,vehicle acceleration, steering, and/or the like) through an experimentbased on a condition of an experimented driver and is stored in thememory, and may be varied (W_(d)∝W_(l)) depending on a drivingcondition. That is, a condition expressed as the following Equation (12)may be satisfied for determining a driving status of the driver as thesafety driving status:

$\begin{matrix}{{{\sum\limits_{t = {t\; 1}}^{t = {t\; 2}}\left\{ {{W_{d}\left( {t,n} \right)} - \left( {{v(t)} + {T(n)} + {E(t)} + {P(t)}} \right)} \right\}} \geq W_{i}}{W_{d} = {{d_{1}t_{1}} + {d_{2}t_{2}} + {d_{3}t_{3}} + {d_{4}t_{4}}}}} & (12)\end{matrix}$

where t denotes a time value, and n is a times value.

The output unit 1165 may be an element for outputting a screen and awarning sound, and may include a liquid crystal display (LCD) and aspeaker. For example, as illustrated in FIG. 16 (b), the output unit1165 may output the screen and the warning sound through a cluster.Alternatively, the output unit 1165 may display the screen through anaudio display. In addition, as illustrated in (a) of FIG. 21, the outputunit 1165 may display a safety driving status determination result and aresult value thereof, which are obtained by the control unit 1163, onthe screen in a rod graph form. In this case, the rod graph may bedisplayed in various colors.

For example, the control unit 1163 may control the output unit 300 todisplay information of a current load, which is based on a differencebetween the driving load and the driving interruption load, as a rodgraph on a screen.

In detail, a result value “C” which is obtained by dividing, by thespare load “W_(l)”, a difference between the driving load “W_(d)” andthe driving interruption load “W_(i)” calculated by the control unit1163 may be shown as a percentage unit of the rod graph.

For example, a case where it is difficult to drive the vehicle is set to75%, and when the result value “C” is 75% or more, the control unit 1163may control the output 1165 to display the rod graph in red. Also, bycontrolling the output unit 1165 for the rod graph to flicker, thecontrol unit 1163 may remind the driver of a driving danger of thevehicle.

In this case, the control unit 1163 may control the output unit 1165 tooutput the warning sound, thereby issuing a warning to the driver.

When the result value “C” is less than 75%, the rod graph may bedisplayed in green and thus may inform the driver of a safe status.

Furthermore, when the result value “C” which is obtained by dividing adifference between the driving load “W_(d)” and the driving interruptionload “W_(i)” by the spare load “W_(l)” is equal to or greater than acertain level (for example, 85%), the control unit 1163 may forciblyturn off power of an AVN which has no difficulty to perform the safedriving of the vehicle.

In addition, the control unit 1163 may increase volume of the warningsound output by the output unit 1165 so as to be proportional to thepower of the result value “C”, thereby strongly warning the driver of adriving danger.

As described above, the present invention may compare a driving load,which occurs in driving a vehicle, with a driving interruption loadwhich interrupts the driving of the vehicle like a peripheral device,drowsiness of a driver, and/or the like, and thus check a safety drivingstatus of the driver. The present invention may issue a warning to thedriver when a load of interrupting the driving of the vehicle is high,and depending on the case, the present invention may forcibly stop anoperation of an element (for example, a peripheral device), which is notessential for the driving of the vehicle, to decrease a drivinginterruption load which is interrupts the driving of the vehicle,thereby enabling the driver to safely drive the vehicle.

FIG. 22 is a flowchart illustrating a driver status sensing methodaccording to an embodiment of the present invention.

First, in step S2210, the driver status sensing system 1160 may acquiredriving manipulation information and careless status information of adriver.

In detail, the driver status sensing system 1160 may sense drivingmanipulations of driving control units, which are essential for drivingof the vehicle, to acquire the driving manipulation information.

For example, the driver status sensing system 1160 may acquire drivingmanipulation information by sensing at least one of acceleration pedal(ACC pad) manipulation, brake pedal manipulation, multi-function switchmanipulation, and steering handle manipulation. In this case, the driverstatus sensing system 1160 may check a velocity of the vehicle, and whenthe velocity of the vehicle is equal to or higher than a certainvelocity (for example, 10 km/h), the driver status sensing system 1160may acquire the driving manipulation information.

For example, the driver status sensing system 1160 may sense the number“n_(A)” of operations of an acceleration pedal manipulated by a driverat every certain time for a predetermined time. Likewise, the driverstatus sensing system 1160 may sense the numbers “n_(B)” and “n_(C)” ofoperations of the brake pedal and the clutch pedal which are manipulatedat every certain time.

As another example, the driver status sensing system 1160 may sense thenumber “n_(M)” of operations of each multi-function switch which ismanipulated by a driver for a certain time. Here, each multi-functionswitch may be a switch for operating a vehicle wiper or a vehicle lampsuch as a turn signal lamp.

As another example, the driver status sensing system 1160 may sense anangular velocity of a steering wheel which is manipulated by the driverfor a certain time. In this case, the driver status sensing system 1160may measure an angle change amount of the steering wheel to calculatethe angular velocity without separately measuring the angular velocity.

Furthermore, the driver status sensing system 1160 may acquire carelessstatus information, based on a peripheral device which is selectivelycontrolled by the driver while the vehicle is driving, a voice of thedriver, and facial information.

For example, the driver status sensing system 1160 may sense the number“nT” of manipulations of peripheral devices which are made by the driverwhile the vehicle is driving. Here, the peripheral devices may not bedevices essential for driving of the driver unlike an AVN, a vehicle airconditioning device, etc., but may be control units which aremanipulated for convenience of the driver or a vehicle indoorenvironment.

As another example, the driver status sensing system 1160 may sense avoice of the driver through a microphone which is disposed at a certainposition in the vehicle. For example, the driver status sensing system1160 may check a voice production time and a pulse level (a voice level)of voice data which is received from the microphone for a certain time.

As another example, the driver status sensing system 1160 may receive afacial image of the driver from a camera, which is disposed at a certainposition of the vehicle, to acquire eye-closing information andobservation negligence information of the driver. In this case, asillustrated in FIG. 16, the camera may include a near infrared LED forrecording an image at daytime and nighttime.

For example, the driver status sensing system 1160 may separatelyextract an eye region image of the driver from the facial image of thedriver. In this case, the eye region image may be extracted from thefacial image of the driver through image processing. That is, the driverstatus sensing system 1160 may acquire careless status information suchas the eye closing and observation negligence of the driver, based onthe facial image of the driver acquired from the camera and the eyeregion image included in the facial image.

The driver status sensing system 1160 may acquire the eye-closinginformation of the driver, based on the eye region image extracted fromthe facial image of the driver. Referring to FIG. 17, the driver statussensing system 1160 may sense an eyelid region from the eye regionimage, and when a sum of ∠A and ∠B which are angles of eyelids is equalto or less than ∠C which is a predetermined threshold angle value(∠A+∠B≤∠C), the careless status sensing unit 1161B may determine thedriver as closing eyes.

The driver status sensing system 1160 may check the angles of theeyelids for a certain time to sense the number of times the drivercloses eyes, and calculate a certain time value and the number of timesthe driver closes the eyes, thereby acquiring eye-closing information (adrowsiness time) of the driver.

Moreover, the driver status sensing system 1160 may acquire theobservation negligence of the driver, based on the facial image of thedriver and the eye region image included in the facial image.

Referring to FIG. 20, in an observation negligence range, observationnegligence information may be acquired by checking (a viewing direction)whether a viewing range of the driver is within a visual distance “η”where there is no load when the vehicle drives, based on an angle of avehicle wheel (i.e., an angle change amount (ΘA or ΘB in (a) of FIG. 20)of a wheel in a center of the vehicle) instead of an angle of a steeringwheel. The viewing range may be checked by performing an operation whichcalculates an angle of a face (α in (b) of FIG. 20) from the facialimage of the driver acquired from the camera and then measures aposition (a position of pupil) of a pupil (β in (c) of FIG. 20) in theeye region image.

In detail, in a case where a gear lever is D gear or N gear when anangle of the steering wheel is within a certain angle, when the viewingof the driver does not enter a driving non-load visual range (a, b) fora certain time, the driver status sensing system 1160 may determineobservation negligence, check a corresponding time, and acquire theobservation negligence information.

First, in step S2220, the driver status sensing system 1160 maycalculate the driving manipulation load and the driving interruptionload, based on the driving manipulation information and the carelessstatus information which is acquired in step S2210.

For example, the driver status sensing system 1160 may calculate thedriving manipulation load “W_(d)”, based on the driving manipulationinformation. In detail, the driver status sensing system 1160 mayacquire the driving manipulation information, read an item-based weightvalue of the driving manipulation information from the memory, andcalculate the driving manipulation load.

Here, the weight value may be a value which is extracted andpredetermined for each item of driving manipulation information throughvarious experiments. Alternatively, the weight value may be a valuewhich is arbitrarily predetermined for each item by a worker.

Moreover, the driver status sensing system 1160 may calculate a load foreach item, based on an item included in the driving manipulationinformation and a weight value corresponding to the item. For example,when driving manipulation information of acceleration pedalmanipulation, brake pedal manipulation, multi-function switchmanipulation, and steering handle manipulation is acquired, the driverstatus sensing system 1160 may read, from the memory, a weight value“D_(A)” of the acceleration pedal manipulation, a weight value “D_(B)”of the brake pedal manipulation, a weight value “D_(M)” of themulti-function switch manipulation, and a weight value “θ” of thesteering handle manipulation. For example, when time information andtimes information of each item are acquired in units of 50 ms for secondof 200 ms, the driving manipulation load “W_(d)” for second of 200 msmay be expressed as Equation (10).

As another example, the driver status sensing system 1160 may calculatethe driving interruption load “W_(i)”, based on the careless statusinformation. In detail, the driver status sensing system 1160 maycalculate loads of the items included in the careless status informationand summate the calculated loads of the items to calculate the drivinginterruption load.

The driver status sensing system 1160 may calculate a peripheral devicemanipulation load “T(n)”, based on the number of manipulations ofperipheral devices. For example, the driver status sensing system 1160may calculate the peripheral device manipulation load in furtherconsideration of a weight value of peripheral device manipulation whichis stored in the memory. For example, when the number of inputs ofperipheral device manipulation which is made by the driver for second of20 ms is acquired, the driver status sensing system 1160 may perform anarithmetic operation on the number of inputs of the peripheral devicemanipulation and a weight value to calculate the peripheral devicemanipulation load “T(n)”.

Furthermore, the driver status sensing system 1160 may calculate a voiceload “V(n)”, based on voice data of the driver acquired. In detail, thedriver status sensing system 1160 may calculate the voice load, based ona pulse threshold value of the voice data stored in the memory. Forexample, the driver status sensing system 1160 may calculate a time, atwhich voice data having the pulse threshold value or more is received,in the voice data of the driver which is acquired for a certain time(for example, 200 ms), thereby calculating a voice load “V(t)”.

Moreover, the driver status sensing system 1160 may calculate adrowsiness load “P(t)” and a driving observation negligence load “E(t)”,based on the eye-closing information and the observation negligenceinformation which are included in the careless status information.

In this case, the driver status sensing system 1160 may perform anoperation of checking the eye closing of the driver as in the flowchartof FIG. 19. The driver status sensing system 1160 may check the number“y” of eye closings from an angle of an eyelid per certain time “x” fora certain measurement time “N” to calculate an eye closing time (adrowsing time) of the driver. Furthermore, the driver status sensingsystem 1160 may convert a value, which is obtained by counting (P) acase where a PERCLOS value is equal to or more than 30% for a certaintime, into a load factor to calculate a drowsiness load “P(t)”.

Moreover, the driver status sensing system 1160 may calculate theobservation negligence load “E(t)”, based on the observation negligenceinformation included in the careless status information. As illustratedin FIG. 20, when the viewing range deviates from the driving non-loadvisual distance “η”, a factor value of the load factor “P” may bechanged according to a deviating range. In FIG. 20, when a deviatingrange (a′, b′) is predetermined, the driver status sensing system 1160may read a factor value of a load factor from the memory for each rangecorresponding to a current viewing range of the driver, therebycalculating the observation negligence load “E(t)”.

The driver status sensing system 1160 may perform an arithmeticoperation on the calculated peripheral device manipulation load “T(n)”,voice load “V(t)”, drowsiness load “P(t)”, and observation negligenceload “E(t)” to calculate the driving interruption load “W_(i)”.

In step S2230, the driver status sensing system 1160 may compare apredetermined spare load “W_(l)” with a difference between the drivingload “W_(d)” and the driving interruption load “W_(i)”.

In detail, when a value of the driving interruption load “W_(i)”subtracted from the driving load “W_(d)” is equal to or less than apredetermined spare load “W_(l)”, the driver status sensing system 1160may determine that the driver is not in a safety driving status. Here,the predetermined spare load “W_(l)” may be a value which is extractedfrom a driving load and driving information (including a change gearstatus, vehicle acceleration, steering, and/or the like) through anexperiment based on a condition of an experimented driver and is storedin the memory, and may be varied (W_(d)∝W_(l)) depending on a drivingcondition. In this case, the driver status sensing system 1160 maycalculate a result value “C” which is obtained by dividing thedifference between the driving load “W_(d)” and the driving interruptionload “W_(i)” by the predetermined spare load “W_(l)”.

For example, a case where it is difficult to drive the vehicle may beset to a threshold percentage value, and when the result value “C” whichis calculated in step S2230 is equal to or greater than the thresholdpercentage value, the driver status sensing system 1160 may issue awarning to the driver in step S2240.

The case where it is difficult to drive the vehicle is set to 75%, andwhen the result value “C” is 75% or more, the driver status sensingsystem 1160 may display a rod graph, which shows a current load, in red.Also, by allowing the rod graph to flicker, the driver status sensingsystem 1160 may remind the driver of a driving danger of the vehicle. Inthis case, the driver status sensing system 1160 may output a warningsound to issue a warning to the driver.

For example, when the result value “C” which is calculated in step S2230is less than 75%, the driver status sensing system 1160 may display therod graph, which shows the current load, in green and thus may informthe driver of a safe status in step S2250.

As described above, the present invention may compare a driving load,which occurs in driving a vehicle, with a driving interruption loadwhich interrupts the driving of the vehicle like a peripheral device,drowsiness of a driver, and/or the like, and thus check a safety drivingstatus of the driver. The present invention may issue a warning to thedriver when a load of interrupting the driving of the vehicle is high,and depending on the case, the present invention may forcibly stop anoperation of an element (for example, a peripheral device), which is notessential for the driving of the vehicle, to decrease a drivinginterruption load which interrupts the driving of the vehicle, therebyenabling the driver to safely drive the vehicle.

Moreover, when a driving habit indicates a case where a vehicle isdriven by a plurality of different drivers, learning is performedwithout considering that the different drivers have different drivingpattern learnings and driving situations, thereby removing a driverhabit recognition defect.

Moreover, the present invention solves a problem where lanes are notrecognized in lateral control when backlight occurs and a vehicle entersa tunnel, thereby enlarging a lateral controller available range.

Moreover, the present invention may be applied to a self-driving vehiclecontroller which will be developed later.

FIG. 23 is a block diagram schematically illustrating a situationdetection apparatus according to an embodiment of the present invention.FIG. 24 is an illustrative view of a driver status detection section ofFIG. 23. FIG. 25 is an illustrative view of a vehicle surroundingsituation detection section of FIG. 23. FIG. 26 is a block diagramillustrating a determination unit of FIG. 23. FIG. 27 is a block diagramillustrating a warning unit of FIG. 23. Referring to FIGS. 23 to 27, thesituation detection apparatus according to the embodiment of the presentinvention includes a detection unit 2010, a driving pattern learningunit 2020, a weighting determination unit 2030, a determination unit2040, a warning unit 2050, and a memory unit 2060. The detection unit2010, the driving pattern learning unit 2020, the weightingdetermination unit 2030, the determination unit 2040, the warning unit2050, and the memory unit 2060 may also be interconnected in a wirelessmanner using Bluetooth, ZigBee, WiFi, etc. or in a wired manner usingRS-232, RS-485, CAN, etc.

A driver status detection section 2011 is a component to acquire vehicledriving information, vehicle operation information, and driver statusinformation. The vehicle driving information means, for example,information such as how often a driver steps on an accelerator pedal,how often a driver steps on a brake pedal, how often a driver operates asteering wheel, and how often a driver operates a multifunctionalswitch. In addition, in a manual transmission vehicle, the drivinginformation may include information such as how often a driver steps ona clutch pedal and how often a driver operates a transmission, besidesthe above information. The multifunctional switch means a switch of awiper, a turn signal indicator, a lighting lamp, or the like. Since themultifunctional switch is a factor necessary to vehicle driving,operation information of the multifunctional switch may be included inthe vehicle driving information. The vehicle operation information mayinclude, for example, information such as how often a driver operates anAVN (Audio Video Navigation) and how often a driver operates an airconditioning device. The driver status information may include, forexample, information such as how long a driver makes conversation(including a telephone conversation), whether or not a driver drowses,whether or not a driver keeps eyes forward, and whether or notabnormality is generated in a driver's electrocardiogram or brainwave.The detection unit 2010 is a component to identify a driver and acquiredriver status data, vehicle driving information data and vehiclesurrounding obstacle data. The detection unit 2010 includes a driverstatus detection section 2011 and a vehicle surrounding situationdetection section 2012.

The driver status detection section 2011 may include one or more of aninfrared LED imaging device, a steering wheel speed detection sensor, asteering wheel angle detection sensor, a suspension movement detectionsensor, a pedal operation detection sensor, a multifunctional switchoperation detection sensor, a voice recognition sensor, an AVN operationdetection sensor, an air conditioning device operation detection sensor,a gearbox sensor, a console box operation detection sensor, and a glovebox operation detection sensor. The driver status detection section 2011also acquires information about driver operation and behavior, which arenot directly related to driving, so as to provide reasons for situationdetermination. The driver status detection section 2011 acquires imagedata of pupils and faces through the infrared LED imaging device. Thus,the current driver is identified. The driver status detection section2011 acquires eyelid detection data to determine whether or not a driverdrowses and pupil direction data to determine whether or not a driverkeeps eyes forward through the infrared LED imaging device. In addition,the driver status detection section 2011 acquires data generated byoperations of an accelerator pedal, a brake pedal, a steering wheel, amultifunctional switch, etc. The driver status detection section 2011acquires data about how long a driver makes conversation (including atelephone conversation) by recognizing a voice of the driver. The driverstatus detection section 2011 acquires data about how often a driveroperates peripheral devices such as an AVN, an air conditioning device,a gearbox, a console box, and a glove box.

The vehicle surrounding situation detection section 2012 is a componentto acquire self-vehicle driving information and surrounding vehicledriving information. The self-vehicle driving information means, forexample, information such as a self-vehicle speed, a yaw rate, asteering angle, an acceleration, a steering wheel angle change amount,and an angular speed in a self-vehicle. The surrounding vehicle drivinginformation means, for example, information such as a surroundingvehicle speed, a yaw rate, a steering angle, an acceleration, a steeringwheel angle change amount, and an angular speed in a surroundingvehicle. To this end, the vehicle surrounding situation detectionsection 2012 may include one or more of a self-vehicle speed sensor, ayaw rate sensor, a steering angle sensor, an acceleration sensor, asteering wheel sensor, front/rear radars, front/rear cameras, sideultrasonic devices, an AVM (Around View Monitoring System) camera, anSCC (Smart Cruise Control), an LKAS (Lane Keeping Assistant System), anSPAS (Smart Parking Assistant System), and an AVM (Around ViewMonitoring). The vehicle surrounding situation detection section 2012collects the self-vehicle driving information and the surroundingvehicle driving information such as surrounding obstacles andsurrounding environments, thereby enhancing reliability when a degree ofrisk is determined during driver's driving.

The driving pattern learning unit 2020 learns a driver's driving pattern(updates data) and stores learned data in the memory unit 2060, based onthe data acquired by the detection unit 2010.

In a case of a new driver, the driving pattern learning unit 2020 callsa pre-stored default driving pattern to execute situation detection andnewly allocates memory to begin learning. The pre-stored default drivingpattern may be an average driving pattern of a plurality of driversdefined by experiment. Thus, it may be possible to induce the new driverto drive the vehicle with safety corresponding to the driving of the newdriver, unlike the related art. The driving pattern learning unit 2020learns and stores a driving pattern within a preset learning range. Apossibility of unnecessary warning is increased when a driving patternsuch as rapid acceleration, rapid brake, or rapid rotation according tounexpected situations during driving is learned under the samecondition. Accordingly, in order to remove such noise, a drivingpattern, which is a subject to be learned, is restricted to the drivingpattern within a preset learning range. The preset range may bedetermined by experiment according to conditions of test subjects.

The weighting determination unit 2030 determines a weighting assigned tothe information data acquired by the detection unit 2010, based on thedriving pattern learned by the driving pattern learning unit 2020. Thisis to induce safe driving by providing a situation detection and warningsystem specified for each driver according to the learned drivingpattern. For example, when a driver A frequently operates theaccelerator pedal, the brake pedal, the multifunctional switch, or thelike, a weighting of operation information data of such a device is sethigh, and when a driver B frequently operates the AVN switch, the airconditioning device, or the like, a weighting of operation informationdata of such devices is set high. The weighting may be determined byexperiment according to conditions of test subjects. Thus, a weightingis also determined about the new driver and thus the situation detectionof the vehicle may be performed. The determined weighting is changed asin the following equation, by comparing acquired information data and acalculated integrated risk index with an information data referencevalue and an integrated risk index reference value, by feedback from thewarning unit 2050. Thus, it may be possible to induce safe driving byproviding the situation detection and warning system specified for eachdriver according to the driver's driving pattern.α(n+1)=α(n), if W<d _(W) and R≥d _(R)α(n+1)=α(n)+δ, if W≥d _(W)α(n+1)=α(n)−δ, if W<d _(W) and R<d _(R),

-   -   where R=integrated risk index,    -   α=weighting,    -   α(n+1)=weighting changed by feedback,    -   α(n)=weighting before change,    -   W=acquired information data,    -   d_(R)=integrated risk index reference value,    -   d_(W)=information data reference value, and    -   δ=α(n)/R.

In the above equation, d_(W) refers to an information data referencevalue and d_(R) refers to an integrated risk index reference value. Theweighting determination unit 2030 compares current R and W values withrespective reference values and changes the weighting as in FIG. 30 andthe above equation. In addition, α(n+1) refers to a weighting changed byfeedback and α(n) refers to a weighting before change. In addition, δrefers to a rate occupied by a weighting before change of eachinformation data in the integrated risk index and the weighting of eachinformation data may be increased or decreased by a δ value.

The determination unit 2040 is a component which determines a safedriving state of a driver, based on the data to which the weightingdetermined by the weighting determination unit 2030 is assigned. Thedetermination unit 2040 includes a calculation learning unit 2070, acalculation unit 2080, an examination unit 2090, and a control unit2100.

The calculation learning unit 2070 arranges data, to which the weightingdetermined by the weighting determination unit 2030 is assigned, in theorder of data causing the integrated risk index to exceed a presetreference risk index and selects only a plurality of high data. Thesituation detection apparatus of the present invention collects a greatdeal of data from a plurality of sensors inside/outside the vehicle toaccurately detect situations. However, when a considerable time isrequired in processing a great deal of data, it is deviated from thepurpose of the present invention for preventing accidents and inducingsafe driving. Accordingly, there is a need to select some data from agreat deal of data and perform rapid calculation during driving requiredfor instantaneous determination. Thus, learning for rapid calculationand data selection are performed in the calculation learning unit 2070.A selection function of the calculation learning unit 2070 is initiallyinactivated and the calculation learning unit 2070 stores resultsarising from the control unit 2100. Subsequently, when data equal to ormore than a certain number of times are stored, the calculation learningunit 2070 arranges the data in the order of data mainly causing theintegrated risk index calculated by the calculation unit 2080 to exceeda preset reference risk index and selects only a plurality of high data.Here, the selected data is used for calculation and the remaining datais ignored such that calculation speed is increased.

For example, when risk warnings equal to or more than 5000 times aregenerated and risk index data equal to or more than 5000 times arestored, data causing the calculated integrated risk index to exceed apreset reference risk index is selected. That is, the highest three datamay be selected from data to which the weighting is reflected byanalysis of causes such as the number of times of operation of the brakepedal, a steering wheel angle change amount, a forward observationneglect of a driver, and a trajectory of a surrounding vehicle.

Subsequently, calculation speed may be increased in such a manner thatonly the selected data is reflected to calculate the integrated riskindex and the remaining data is not reflected to calculate theintegrated risk index. The data selected by the calculation learningunit 2070 is transferred to the calculation unit 2080.

The calculation unit 2080 calculates some data selected by thecalculation learning unit 2070 among data, to which the weightingdetermined by the weighting determination unit 2030 is assigned,according to a preset calculation equation and calculates an integratedrisk index. For example, a driver's integrated risk index may becalculated by adding up respective risk indexes which multiply dataselected by the calculation learning unit 2070 by a weighting for eachdata assigned by the weighting determination unit, as in the followingequation.R=α _(A) ×W _(A)+α_(B) ×W _(B)+α_(C) ×W _(C),

-   -   where R=integrated risk index,    -   α=weighting for each selected information data,    -   W=selected information data.

The examination unit 2090 determines whether a result calculated by thecalculation unit 2080 is valid. When the calculation learning unit 2070selects some data for rapid calculation and calculation is performedbased on the same, there is a possibility of error occurring in thecalculated result. Since the object of the present invention is toinduce safe driving, an examination process for error removal isrequired for driver's safe driving. The examination unit 2090 compares arisk index calculated by a pre-stored driving pattern of a currentdriver (hereinafter, referred to as “examination risk index) with anintegrated risk index calculated by a driving pattern learning a currentdriving pattern of the current driver and determines that a calculatedrisk index value is valid when a difference between the examination riskindex and the calculated integrated risk index is within a preset errorrange. In addition, the integrated risk index is transferred to thecontrol unit 2100. The preset error range may be an experimental valueaccording to conditions of test subjects. In addition, the examinationrisk index may be previously calculated before current driving of thedriver and stored in the memory unit 2060.

The control unit 2100 compares the integrated risk index transferredfrom the examination unit 2090 with a preset reference risk index andserves to activate the warning unit 2050 according to the comparedresult.

The warning unit 2050 is a component which warns a driver that thedriving state of the driver is not in a safe driving state to induce thesafe driving by the determination unit 2040, and includes a warningsound output device 2051, a driving load display device 2052, and avehicle control device 2053. When the driving state of the driver isdetermined to be not in the safe driving state by the determination unit2040, the warning sound output device 2051 may generate a warning soundto the driver or play an announcement for notifying that the driver isnot in the safe driving state. The warning sound output device 2051 mayalso utilize a speaker installed to the vehicle. The driving loaddisplay device 2052 may also display a driving load through aninstrument panel, an AVN, or an HUD (Head Up Display) in the vehicle.The vehicle control device 2053 is a device to safely stop the vehiclewhen the driver is determined to be not in the safe driving state, andmay be a device for controlling a steering wheel, a transmission, and abrake which are installed to the vehicle. The memory unit 2060 may storeand call information such as driver information, driver's drivingpattern information, a preset weighting, a preset error range, apre-stored examination risk index, and a preset reference risk index.The memory unit 2060 may be a nonvolatile memory as a storage means forstoring data. For example, the driver information of a driver A, adriver B, a driver C, . . . , etc. is stored in the memory unit 2060 andthe driving pattern information corresponding to each driver is storedin the form of “driver A—driving pattern A”, “driver B—driving patternB” “driver C—driving pattern C”, “driver A-driving pattern A”, . . . ,etc. in the memory unit 2060. In this case, the data acquired by thedetection unit 2010, such as the number of times of operation of theaccelerator pedal for unit time and the number of times of operation ofthe brake pedal for unit time, are included in each driving pattern.

FIG. 28 is a flowchart schematically illustrating a situation detectionmethod according to another embodiment of the present invention. FIG. 29is a flowchart illustrating a driving pattern learning step. FIG. 30 isa view illustrating a state in a weighting determination step. FIG. 31is a flowchart illustrating a calculation learning step. FIG. 32 is aflowchart illustrating an examination step. FIG. 33 is a flowchartillustrating a warning step. FIGS. 34 to 36 are detailed flowchartsillustrating the situation detection method. Referring to FIGS. 28 to36, the situation detection method according to another embodiment ofthe present invention includes a driver recognition step S2810, S2910,S3110, S3410, a driving pattern calling step S2820, S2920, S3120, S3420,a detection step S2830, S3430, a driving pattern learning step S2840,S2940, S3450, a weighting determination step S2850, S3550, adetermination step S2860, S3560, and a warning step S2870, S3370, S3670.

The driver recognition step S2810 is a step of calling pre-stored driverinformation to compare whether the driver information coincides with acurrent driver, and includes a driver data acquisition step S3411 and adriver identification step S3412. The driver data acquisition step S3411is a step of acquiring image data of pupils or faces through an imagingdevice and the driver identification step S3412 is a step of comparingthe image data of pupils or faces acquired by the driver dataacquisition step S3411 with the driver information pre-stored in amemory unit 2060 to identify a driver.

The driving pattern calling step S2820 is a step of calling a pre-storeddriving pattern of the driver identified by the driver identificationstep S2810 from the memory unit 2060. If the driver coinciding with thepre-stored driver information is not present because of a new driver, apre-stored default driving pattern is called. The pre-stored defaultdriving pattern may be an average driving pattern of a plurality ofdrivers defined by experiment. Thus, it may be possible to induce thenew driver to drive a vehicle with safety corresponding to the drivingof the new driver even in a case of the new driver, unlike the relatedart.

The detection step S2830 is a step of collecting driver statusinformation and vehicle driving information or vehicle surroundingobstacle information, and includes a driver status detection step S2831and a vehicle surrounding situation detection step S2832.

The driver status detection step S2831 is a step of collecting driverstatus information and acquires vehicle driving information, vehicleoperation information, and driver status information. The vehicledriving information means, for example, information such as how often adriver steps on an accelerator pedal, how often a driver steps on abrake pedal, how often a driver operates a steering wheel, and how oftena driver operates a multifunctional switch. In addition, in a manualtransmission vehicle, the driving information may include informationsuch as how often a driver steps on a clutch pedal and how often adriver operates a transmission, besides the above information. Themultifunctional switch means a switch of a wiper, a turn signalindicator, a lighting lamp, or the like. Since the multifunctionalswitch is a factor necessary to vehicle driving, operation informationof the multifunctional switch may be included in the vehicle drivinginformation.

The vehicle operation information may include, for example, informationsuch as how often a driver operates an AVN (Audio Video Navigation), howoften a driver operates an air conditioning device, and how often adriver operates peripheral devices such as a gearbox, a console box, anda glove box. The driver status information may include, for example,information such as how long a driver makes conversation (including atelephone conversation), whether or not a driver drowses, whether or nota driver keeps eyes forward, and whether or not abnormality is generatedin a driver's electrocardiogram or brainwave.

To this end, the driver status detection step S2831 may detect one ormore of a driver eyelid, a driver pupils, a steering wheel speed, asteering wheel angle, a suspension movement, whether or not anaccelerator pedal is operated, whether or not a brake pedal is operated,whether or not a multifunctional switch is operated, whether or not adriver makes conversation, whether or not an AVN is operated, whether ornot an air conditioning device is operated, whether or not a gearbox isoperated, whether or not a console box is operated, and whether or not aglove box is operated.

The vehicle surrounding situation detection step S2832 acquiresself-vehicle driving information, surrounding vehicle drivinginformation, vehicle surrounding obstacle information, etc. Theself-vehicle driving information means, for example, information such asa self-vehicle speed, a yaw rate, a steering angle, an acceleration, asteering wheel angle change amount, and an angular speed in aself-vehicle. The surrounding vehicle driving information means, forexample, information such as a surrounding vehicle speed, a yaw rate, asteering angle, an acceleration, a steering wheel angle change amount,and an angular speed in a surrounding vehicle. The vehicle surroundingobstacle information means, for example, information such as a trafficsituation of a forward road, a road shape, and a road surface state. Thevehicle surrounding situation detection step S2832 detects a vehiclesurrounding situation using one or more of an SCC (Smart CruiseControl), an LKAS (Lane Keeping Assistant System), an SPAS (SmartParking Assistant System), an AVM (Around View Monitoring), a camera,and a radar. The vehicle surrounding situation detection step S2832collects the self-vehicle driving information and the surroundingvehicle driving information such as surrounding obstacles andsurrounding environments, thereby enhancing reliability when a degree ofrisk is determined during driver's driving.

The driving pattern learning step S2840 is a step of learning apre-stored driving pattern of the driver called by the driving patterncalling step S2820 and a driving pattern by the data acquired from thedetection step S2830 to store the learned driving pattern in the memoryunit 2060, and includes a driving pattern comparison step S2941, adriving pattern determination step S2942, a driving pattern storage stepS2943, and a noise removal step S2944. In a case of the new driver, thedriving pattern learning step S2840 calls a pre-stored default drivingpattern to execute situation detection and newly allocates memory tobegin learning. The pre-stored default driving pattern may be an averagedriving pattern of a plurality of drivers defined by experiment. Thus,it may be possible to induce the new driver to drive the vehicle withsafety corresponding to the driving of the new driver, unlike therelated art. In addition, a specific driving pattern for each driver maybe grasped by performing of the driving pattern learning step S2840 andthus a situation detection method specified for each driver may beprovided to induce safe driving.

The driving pattern comparison step S2941 is a step of comparing apre-stored driving pattern of the driver called by the driving patterncalling step S2820, S2920 and a current driving pattern by the dataacquired from the detection step S2830. The driving patterndetermination step S2942 is a step of determining whether a differenceof both in the driving pattern comparison step S2941 is within a presetnoise range. The driving pattern storage step S2943 is a step oflearning a current driving pattern when the difference of both is equalto or less than the preset noise range in the driving patterndetermination step S2942 to store the learned driving pattern in thememory unit 2060. The noise removal step S2944 is a step of excluding acurrent driving pattern from a subject to be learned when the differenceof both exceeds the preset noise range in the driving patterndetermination step S2942. The preset noise range may be an experimentalvalue according to conditions of test subjects.

The driving pattern learning step S2840, S2940 learns and stores adriving pattern within a preset noise range. A possibility ofunnecessary warning is increased when a driving pattern such as rapidacceleration, rapid brake, or rapid rotation according to unexpectedsituations during driving is learned under the same condition.Accordingly, the driving pattern learning step S2840, S2940 is to removesuch noise. The preset noise range may be determined by experimentaccording to conditions of test subjects.

The weighting determination step S2850 is a step of determining aweighting assigned to each data acquired from the detection step S2830,based on the driving pattern learned in the driving pattern learningstep S2840. This is to induce safe driving by providing a situationdetection and warning system specified for each driver according to thelearned driving pattern. For example, when a driver A frequentlyoperates the accelerator pedal, the brake pedal, the multifunctionalswitch, or the like, a weighting of operation information data of such adevice is set high, and when a driver B frequently operates the AVNswitch, the air conditioning device, or the like, a weighting ofoperation information data of such devices is set high.

The weighting may be determined by experiment according to conditions oftest subjects. Thus, a weighting is also determined about the new driverand thus the situation detection of the vehicle may be performed. Thedetermined weighting is changed as in the following equation by feedbackfrom an information transfer step S3374 of the warning step S2870,S3370. Thus, it may be possible to induce safe driving by providing thesituation detection and warning system specified for each driveraccording to the driver's driving pattern.α(n+1)=α(n), if W<d _(W) and R≥d _(R)α(n+1)=α(n)+δ, if W≥d _(W)α(n+1)=α(n)−δ, if W<d _(W) and R<d _(R),

-   -   where R=integrated risk index,    -   α=weighting,    -   α(n+1)=weighting changed by feedback,    -   α(n)=weighting before change,    -   W=acquired information data,    -   d_(R)=integrated risk index reference value,    -   d_(W)=information data reference value, and    -   δ=α(n)/R.

In the above equation, d_(W) refers to an information data referencevalue and d_(R) refers to an integrated risk index reference value. Theweighting determination step S2850 compares current R and W values withrespective reference values and changes the weighting as in FIG. 30 andthe above equation. In addition, α(n+1) refers to a weighting changed byfeedback and α(n) refers to a weighting before change. In addition, δrefers to a rate occupied by a weighting before change of eachinformation data in the integrated risk index and the weighting of eachinformation data may be increased or decreased by a δ value.

The determination step S2860, S3560 is a step of determining a safedriving state of a driver, based on the data to which the weightingdetermined in the weighting determination step S2850, S3550 is assigned.The determination step S2860, S3560 includes a calculation learning stepS3561, a calculation step S3562, an examination step S3563, and acontrol step S3564.

The calculation learning step S3561 is a step of arranging data, towhich the weighting determined in the weighting determination step S3550is assigned, in the order of data mainly causing the integrated riskindex to exceed a preset reference risk index and of selecting only aplurality of high data. The situation detection method of the presentinvention collects a great deal of data from a plurality of sensorsinside/outside the vehicle to accurately detect situations. However,when a considerable time is required in processing a great deal of data,it is deviated from the purpose of the present invention for preventingaccidents and inducing safe driving. Accordingly, there is a need toselect some data from a great deal of data and perform rapid calculationduring driving required for instantaneous determination. Thus, learningfor rapid calculation and data selection are performed in thecalculation learning step S3561. A selection function in the calculationlearning step S3561 is initially inactivated and the calculationlearning step S3561 stores results arising from the control step S3564.Subsequently, when data equal to or more than a certain number of timesare stored, the calculation learning step S3561 arranges the data in theorder of data mainly causing the integrated risk index calculated in thecalculation step S3562 to exceed a preset reference risk index andselects only a plurality of high data. Here, the selected data is usedfor calculation and the remaining data is ignored such that calculationspeed is increased. For example, when risk warnings equal to or morethan 5000 times are generated and risk index data equal to or more than5000 times are stored, data causing the calculated integrated risk indexto exceed a preset reference risk index is selected. That is, thehighest three data may be selected from data to which the weighting isreflected by analysis of causes such as the number of times of operationof the brake pedal, a steering wheel angle change amount, a forwardobservation neglect of a driver, and a trajectory of a surroundingvehicle. Subsequently, calculation speed may be increased in such amanner that only the selected data is reflected to calculate theintegrated risk index and the remaining data is not reflected tocalculate the integrated risk index. The data selected in thecalculation learning step S3561 is transferred to the calculation stepS3562.

The calculation step S3562 is a step of calculating a driver'sintegrated risk index by adding up respective risk indexes whichmultiply data selected in the calculation learning step S3561 amongdata, to which the weighting determined in the weighting determinationstep S3550 is assign, by a weighting for each data assigned in theweighting determination step S3550. For example, the risk index may becalculated as in the following equation.R=α _(A) ×W _(A)+α_(B) ×W _(B)+α_(C) ×W _(C),

-   -   where R=integrated risk index,    -   α=weighting for each selected information data,    -   W=selected information data.

The examination step S3563 is a step of comparing the integrated riskindex calculated by the calculation step S3562 with a risk indexcalculated based on a pre-stored driving pattern (hereinafter, referredto as “examination risk index) to determine whether the compared resultis within a preset error range. That is, the examination step S3563 is astep of determining whether a result calculated by the calculation stepS3562 is valid. When the calculation learning step S3561 selects somedata for rapid calculation and calculation is performed based on thesame, there is a possibility of error occurring in the calculatedresult. Since the object of the present invention is to induce safedriving, an examination process for error removal is required fordriver's safe driving. Accordingly, the examination step S3563 comparesa risk index calculated by a pre-stored driving pattern of a currentdriver (hereinafter, referred to as “examination risk index) with anintegrated risk index calculated by a driving pattern learning a currentdriving pattern of the current driver and determines that a calculatedrisk index value is valid when a difference between the examination riskindex and the calculated integrated risk index is within a preset errorrange. In addition, the calculated integrated risk index is transferredto the control step S3564. The preset error range may be determined byexperiment according to conditions of test subjects. The examinationrisk index may be previously calculated before current driving of thedriver and stored in the memory unit 2060. On the other hand, since thecalculated integrated risk index is invalid when the difference betweenthe examination risk index and the calculated integrated risk indexexceeds the preset error range, the result is ignored.

The control step S3564 is a step of comparing, when the differencebetween the examination risk index and the calculated integrated riskindex is determined to be within the preset error range in theexamination step S3563, the calculated integrated risk index with apreset reference risk index to determine the compared result. That is,the control step S3564 is a step of comparing the integrated risk indextransferred from the examination step S3563 with a preset reference riskindex and activating the warning step S2870, S3670 according to thecompared result.

The preset reference risk index may include a first reference riskindex, a second reference risk index, and a third reference risk index.Each reference risk index may be determined by experiment according toconditions of test subjects.

When the calculated integrated risk index is equal to or greater than apreset first reference risk index and is less than a preset secondreference risk index, the control step S3564 transfers a signal allowinga first warning step S3671 to be performed. In addition, when thecalculated integrated risk index is equal to or greater than a presetsecond reference risk index and is less than a preset third referencerisk index, the control step S3564 transfers a signal allowing a secondwarning step S3672 to be performed. In addition, when the calculatedintegrated risk index is equal to or greater than a preset thirdreference risk index, the control step S3564 transfers a signal allowinga third warning step S3673 to be performed.

The warning step S3670 is a step of warning a driver when the driver isdetermined to be not in a safe driving state in the determination stepS3560, and includes a first warning step S3671, a second warning stepS3672, a third warning step S3673, and an information transfer stepS3674. The warning step S3670 serves to induce safe driving byperforming respective warning steps of different warning levelsdepending on the signals transferred from the control step S3564 toinform of a warning corresponding to the driver status.

The first warning step S3671 is performed when the integrated risk indexcalculated in the control step S3564 is equal to or greater than apreset first reference risk index and is less than a preset secondreference risk index, and includes one or more of a warning soundgeneration step through a speaker, a warning display step through an AVNor a HUD, and a vibration notification step through vibration of asteering wheel or a seat.

The second warning step S3672 is a step of holding functions of the AVNwhen the integrated risk index calculated in the control step S3564 isequal to or greater than a preset second reference risk index and isless than a preset third reference risk index.

The third warning step S3673 is a step of forcibly stopping a vehiclewhen the integrated risk index calculated in the control step S3564 isequal to or greater than a preset third reference risk index. In thiscase, the vehicle may be forcibly and safely stopped using an ADAS(Advanced Driver Assistance System) module. The ADAS module may includeone or more of an LKAS, an SPAS, and an SCC.

The warning step S3670 includes the information transfer step S3674 oftransferring information to the weighting determining step S3550 forchange of the weighting through feedback. Thus, it may be possible toinduce safe driving by providing the situation detection and warningsystem specified for each driver according to the driver's drivingpattern.

FIG. 37 is a block diagram illustrating a configuration of an apparatusfor detecting a driver status according to an embodiment of the presentinvention. FIG. 38 is a view schematically illustrating a configurationof an information acquisition unit. FIG. 39 is an exemplified viewillustrating an ECG sensor and a PPG sensor. FIG. 40 is an exemplifiedview illustrating an EEG sensor. FIG. 41 is an exemplified viewillustrating a driving load display device according to the embodimentof the present invention. Referring to FIGS. 37 to 41, an apparatus fordetecting a driver status according to an embodiment of the presentinvention includes an information acquisition unit 3010 which acquiresdriver status information, vehicle driving information, and driver'svehicle operation information, a calculation unit 3020 which calculatesa driving load of a driver based on the information acquired by theinformation acquisition unit 3010, a comparison unit 3030 whichcomparing between the driving load calculated by the calculation unit3020 and a preset load margin, and a warning unit 3040 which warns thedriver when the comparison unit 3030 determines that the driving loadexceeds the preset load margin. The information acquisition unit 3010 isa component to acquire the vehicle driving information, the vehicleoperation information, and the driver status information.

The vehicle driving information is information generated when the driverdrives a vehicle, and means, for example, information such as how oftenthe driver steps on an accelerator pedal, how often the driver steps ona brake pedal, how often the driver operates a steering wheel, and howoften the driver operates a multifunctional switch. In addition, in amanual transmission vehicle, the driving information may includeinformation such as how often the driver steps on a clutch pedal and howoften the driver operates a transmission, besides the above information.The multifunctional switch means a switch of a wiper, a turn signalindicator, a lighting lamp, or the like. Since the multifunctionalswitch is a factor necessary to the vehicle driving, operationinformation of the multifunctional switch may be included in the vehicledriving information. The vehicle operation information is informationgenerated when the driver operates the vehicle, and may include, forexample, information such as how often the driver operates an AVN (AudioVideo Navigation) and how often the driver operates an air conditioningdevice. The driver status information is information according to adriver status during driving, and may include, for example, informationsuch as how long the driver makes conversation (including a telephoneconversation), whether or not the driver drowses, whether or not thedriver keeps eyes forward, and whether or not abnormality is generatedin a driver's electrocardiogram or brainwave.

To acquire the above information, the information acquisition unit 3010may include a vehicle driving information acquisition portion 3011, avehicle operation information acquisition portion 3012, and a driverstatus information acquisition portion 3013. The vehicle drivinginformation acquisition portion 3011 may include an accelerator pedalsensor, a brake pedal sensor, a steering wheel sensor, a multifunctionalswitch sensor, a clutch pedal sensor, a transmission sensor. The vehicleoperation information acquisition portion 3012 may include an airconditioning device sensor and an AVN sensor. The driver statusinformation acquisition portion 3013 may include a microphone, a driverobservation camera, an ECG (electrocardiogram) sensor, an EEG(electroencephalogram) sensor, and a PPG (photoplethysmography) sensor.

The microphone is a component to recognize whether or not the drivermakes conversation (including a telephone conversation) and the driverobservation camera is a component to recognize whether or not the driverdrowses or keeps eye forward by capturing a driver's face image or eyearea image. The ECG sensor is a component to recognize a driver'selectrocardiogram and the PPG sensor is a component to recognize adriver's PPG signal. The PPG signal may mean a photoplethysmography. TheECG sensor and the PPG sensor may be a wearable sensor, and,particularly, may have a wearable structure such as a chest belt type ora wristwatch type. The ECG sensor and the PPG sensor may be worn on adriver's body to accurately measure an electrocardiogram and aphotoplethysmography. The EEG sensor is to acquire driver's brainwaveinformation and may be a wearable sensor. Particularly, the EEG sensormay have a wearable structure such as a headset type. The EEG sensor maybe worn on a driver's body to accurately measure a brainwave.

The calculation unit 3020 calculates a driving load indicated byconverting each factor having a negative effect on safe driving of thedriver into a quantitative numerical value, based on the informationacquired by the information acquisition unit 3010. The calculation unit3020 may include a vehicle driving load calculation portion 3021, avehicle operation load calculation portion 3022, and a driver statusload calculation portion 3023. The driving load may be calculated bysumming loads calculated by the respective calculation portions 3021,3022, and 3023.

The comparison unit 3030 comparing between the driving load calculatedby the calculation unit 3020 and a preset load margin. When the drivingload is equal to or less than the preset load margin, the comparisonunit 3030 determines that the driver is in a safe driving state. On theother hand, when the driving load exceeds the preset load margin, thecomparison unit 3030 determines that the driver is not in the safedriving state. The preset load margin may be an experimental valueextracted from a sum of a vehicle driving load, a vehicle operationload, and a driver status load through an experiment according toconditions of a test subject. In addition, the preset load margin may bea value of the driving load calculated based on information according toexisting driving patterns of the driver. The comparison unit 3030 mayinclude a memory portion 3050 for storing a value of the preset loadmargin. The memory portion 3050 may be a nonvolatile memory as a storagemeans for storing data.

The warning unit 3040 is a component to warn the driver when thecomparison unit 3030 determines that the driver is not in the safedriving state, and may include a warning sound output device 3041, adriving load display device 3042, and a vehicle control device 3043. Asshown in FIG. 41, the driving load display device 3042 may be mounted ona dashboard of the vehicle. In addition, the driving load may also bedisplayed through an AVN or a HUD (Head Up Display). When the driver isdetermined to be not in the safe driving state, the warning sound outputdevice 3041 may generate a warning sound to the driver or play anannouncement for notifying that the driver is not in the safe drivingstate. The warning sound output device 3041 may also utilize a speakerinstalled to the vehicle. The vehicle control device 3043 is a device tosafely stop the vehicle when the driver is determined to be not in thesafe driving state, and may be a device for controlling a steeringwheel, a transmission, and a brake which are installed to the vehicle.

The information acquisition unit 3010, the calculation unit 3020, thecomparison unit 3030, the warning unit 3040, and the memory portion 3050may also be interconnected in a wireless manner using Bluetooth, ZigBee,WiFi, etc. or in a wired manner using RS-232, RS-485, CAN, etc.

FIG. 42 is a flowchart schematically illustrating a method of detectinga driver status according to another embodiment of the presentinvention. FIGS. 43 and 44 are detailed flowcharts illustrating aninformation acquisition step. FIGS. 45 and 46 are flowchartsillustrating a calculation step. FIG. 47 is a flowchart illustrating afirst warning step. FIGS. 48 and 49 are flowcharts illustrating themethod of detecting a driver status. FIG. 50 is a view for explaining amethod of determining that the driver closes eyes. FIG. 51 is a view forexplaining a visible range during no-load driving. FIGS. 52 and 53 areviews for explaining a method of determining a driver's viewing range.Referring to FIGS. 42 to 53, a method of detecting a driver statusaccording to another embodiment of the present invention includes aninformation acquisition step S100 which acquires driver statusinformation and driver's vehicle operation information, a calculationstep S200 which calculates a driving load of a driver based on theinformation acquired in the information acquisition step S100, acomparison step S300 which compares the driving load of the drivercalculated in the calculation step S200 and a preset load margin, and awarning step S400 which warns the driver when the comparison step S300determines that the driving load of the driver exceeds the preset loadmargin.

In the information acquisition step S100, information of the driver isacquired by a sensor, a microphone, a camera, etc. The informationacquisition step S100 includes a vehicle driving information acquisitionstep S110 of measuring the number of times the driver operates a pedalor the like for driving the vehicle, an operation time of the pedal orthe like by the driver, etc., a vehicle operation informationacquisition step S120 of measuring the number of times the driveroperates a switch or the like for operating additional devices, anoperation time of the switch or the like by the driver, etc., and adriver status information acquisition step S130 of measuring aconversation time of the driver, an eye-closed time of the driver, atime for which the driver does not keep eyes forward, a driver'sbrainwave, a driver's electrocardiogram, etc.

The vehicle driving information acquisition step S110 includes anaccelerator pedal operation information acquisition step S111 ofmeasuring the number of times the driver operates an accelerator pedalfor a preset unit time, a brake pedal operation information acquisitionstep S112 of measuring the number of times the driver operates a brakepedal for a preset unit time, a steering wheel operation informationacquisition step S113 of measuring an angle change rate of a steeringwheel rotated by the driver for a preset unit time, and amultifunctional switch operation information acquisition step S114 ofmeasuring the number of times the driver operates a multifunctionalswitch such as a wiper or a turn signal indicator for a preset unittime. Particularly, in a manual transmission vehicle, the vehicledriving information acquisition step S110 further includes a clutchpedal operation information acquisition step S115 of measuring thenumber of times the driver operates a clutch pedal for a preset unittime and a transmission operation information acquisition step S116 ofmeasuring the number of times the driver operates a transmission for apreset unit time. The vehicle operation information acquisition stepS120 includes an AVN operation information acquisition step S121 ofmeasuring an operation time of an AVN by the driver and the number oftimes the driver operates the AVN, for a preset unit time, and an airconditioning device operation information acquisition step S122 ofmeasuring an operation time of an air conditioning device such as aheater or an air conditioner by the driver and the number of times thedriver operates the air conditioning device, for a preset unit time.

The driver status information acquisition step S130 includes a driver'svoice information acquisition step S131 of sensing a voice of the driverthrough a microphone mounted at a predetermined position within thevehicle to measure a pulse amplitude (a voice amplitude) of the receivedvoice data and a generation time of the voice having a pulse amplitudeof a reference value or more, a driver's forward observation informationacquisition step S132 of measuring a time for which a driver's viewingrange is deviated from a visible range during no-load driving as arange, in which safe driving is not obstructed, using a driver's faceimage and eye area image captured by a camera mounted at a predeterminedposition within the vehicle, a driver's eye-closed informationacquisition step S133 of measuring the number of times the driver closeseyes and an eye-closed time using a driver's eye area image captured bythe camera mounted at a predetermined position within the vehicle, adriver's brainwave information acquisition step S134, a driver's ECGinformation acquisition step S135, and a PPG signal informationacquisition step S136 of measuring a driver's photoplethysmographicsignal. The respective information acquisition steps are not necessaryto be sequentially performed. For example, the information acquisitionsteps may be simultaneously or reversely performed.

The calculation step S200 includes a vehicle driving load calculationstep S210 of calculating a vehicle driving load indicated by convertingeach factors obstructing safe driving into a quantitative numericalvalue in connection with vehicle driving by the driver, a vehicleoperation load calculation step S220 of calculating a vehicle operationload indicated by converting each factors obstructing safe driving intoa quantitative numerical value in connection with vehicle operation bythe driver, and a driver status load calculation step S230 ofcalculating a driver status load indicated by converting each factorsobstructing safe driving into a quantitative numerical value inconnection with a driver's mental and physical condition, and a drivingload calculation step S240 of calculating a driving load by summing therespective calculated loads.

The vehicle driving load calculation step S210 includes an acceleratorpedal operation load calculation step S211, a brake pedal operation loadcalculation step S212, a multifunctional switch operation loadcalculation step S213, a steering wheel operation load calculation stepS214, and a step S217 of summing the respective calculated operationloads. Since the safe driving may be obstructed when the driverfrequently operates the accelerator pedal, the brake pedal, themultifunctional switch, the steering wheel, etc., the above steps may beincluded in the vehicle driving load calculation step S210. The vehicledriving load calculation step S210 is performed only when a vehiclespeed exceeds a preset speed. In accordance with an exemplary embodimentof the present invention, in a case in which a preset speed is 10 km/h,a vehicle driving load becomes 0 when a vehicle speed by a driver is 9km/h. In the vehicle driving load calculation step S210, the vehicledriving load is calculated by calculating the number of times ofoperation or operation time of each term included in the vehicle drivinginformation acquired in the vehicle driving information acquisition stepS110 and a weighting preset at the term. The preset weighting may be setby an experiment according to each vehicle driving load. In addition,the preset weighting may be a value calculated based on informationaccording to existing driving patterns of the driver. The presetweighting may be stored in the memory portion 3050.

In accordance with another exemplary embodiment of the presentinvention, a vehicle driving load W_(d) is calculated for every 200 ms.Each term is measured in 50 ms and communication is performed in a CANmanner. In a state in which a vehicle starts up, the vehicle drivingload begins to be calculated when a vehicle speed is 10 km/h or more.

(1) When an IG is turned ON, a vehicle speed is 10 km/h or more, a timer1 is set as 200 ms, and a timer 2 is set as 50 ms, whether or not anaccelerator pedal is operated is measured for every 50 ms and a presetaccelerator pedal operation load weighting is loaded from a memory.

(2) When the IG is turned ON, the vehicle speed is 10 km/h or more, thetimer 1 is set as 200 ms, and the timer 2 is set as 50 ms, whether ornot a brake pedal is operated is measured for every 50 ms and a presetbrake pedal operation load weighting is loaded from the memory.

(3) When the IG is turned ON, the vehicle speed is 10 km/h or more, andthe timer 1 is set as 200 ms, whether or not a multifunctional switch isoperated is measured for 200 ms and a preset multifunctional switchoperation load weighting is loaded from the memory.

(4) When the IG is turned ON, the vehicle speed is 10 km/h or more, thetimer 1 is set as 200 ms, and the timer 2 is set as 50 ms, an anglechange rate of a steering wheel is operated is measured for every 50 msand a preset steering wheel operation load weighting is loaded from thememory.

(5) The vehicle driving load W_(d) for 200 ms is calculated according tothe following equation:W _(d) =D _(A) ×n _(A)×50 ms+D _(B) ×n _(B)×50 ms+D _(M) ×n _(M) +θ×n_(θ)×50 ms

-   -   W_(d)=vehicle driving load    -   D_(A)=accelerator pedal operation load weighting    -   n_(A)=number of times of operation of accelerator pedal    -   D_(B)=brake pedal operation load weighting    -   n_(B)=number of times of operation of brake pedal    -   D_(M)=multifunctional switch operation load weighting    -   n_(M)=number of times of operation of multifunctional switch    -   θ=steering wheel operation load weighting    -   n_(θ)=total angle change rate of steering wheel.

(6) Each term included in the vehicle driving information and thevehicle driving load may be added or omitted, if necessary. For example,in the manual transmission vehicle, the vehicle driving load may becalculated by adding a clutch pedal operation load and a transmissionoperation load.

In accordance with still another exemplary embodiment of the presentinvention, a vehicle driving load W_(M) may be calculated by calculatingthe number of times of operation of each term included in the acquiredvehicle operation information and a weighting preset at the term,according to the following equation:

$W_{M} = {\frac{D_{C} \times n_{C} \times T_{C}}{T_{{preset}\mspace{14mu}{time}}} + \frac{D_{C} \times n_{D} \times T_{D}}{T_{{preset}\mspace{14mu}{time}}}}$

W_(M)=vehicle operation load

T_(preset time)=preset time

D_(C)=AVN operation load weighting

n_(C)=number of times of operation of AVN

T_(C)=AVN operation time

D_(D)=air conditioning device operation load weighting

n_(D)=number of times of operation of air conditioning device

T_(D)=air conditioning device operation time.

In accordance with yet another exemplary embodiment of the presentinvention, a driver status load may be calculated by calculating anoperation time of each term included in the acquired driver statusinformation and a weighting preset at the term.

A voice load V may be calculated by sensing a voice of the driverthrough a microphone mounted at a predetermined position within thevehicle and using a pulse amplitude (a voice amplitude) of the receivedvoice data and information of a generation time of the voice, accordingto the following equation:

$V = {\frac{T_{v}}{T_{{preset}\mspace{14mu}{time}}} \times D_{v}}$

V=voice load

T_(preset time)=preset time

T_(V)=generation time of voice having pulse amplitude of reference valueor more

D_(V)=voice load weighting.

Driver's eye-closed information may be acquired using a driver's eyearea image captured by a camera mounted at a predetermined positionwithin the vehicle. The camera may have a near infrared LED to captureimages at the daytime and the nighttime. Referring to FIG. 50, when asum of ∠A and ∠B as angles of an eyelid is equal to or less than ∠C as apreset reference value (∠A+∠B≤∠C) at the time of sensing an eyelid areafrom the eye area image, it is determined that the driver closes eyes. Adrowsiness load may be calculated by identifying an angle of a driver'seyelid for every preset time to sense the number of times the drivercloses eyes and calculating the number of time being eye-closed for thepreset time and an eye-closed time, according to the following equation:

$P = {\frac{T_{p} \times n_{p}}{T_{{preset}\mspace{14mu}{time}}} \times D_{p}}$

P=drowsiness load

T_(preset time)=preset time

T_(P)=eye-closed time

n_(P)=number of time being eye-closed

D_(P)=drowsiness load weighting.

Driver's forward observation information may be acquired using thedriver's face image and eye area image captured by the camera mounted ata predetermined position within the vehicle. The camera may have a nearinfrared LED to capture images at the daytime and the nighttime.Referring to FIG. 51, visible ranges during no-load driving a and b aredetermined based on angles of the wheel. That is, the visible rangesduring no-load driving a and b are determined by angle rates θ_(A) andθ_(B) changed on the basis of a central direction. Referring to FIGS. 52and 53, a driver's viewing range is determined by face angles of thedriver (see FIG. 52) and pupil positions of the driver (see FIG. 53)captured by the camera. An observation neglect load is generated when adriver's current viewing range is deviated from the visible rangesduring no-load driving a and b, and a preset observation neglect loadweighting to may vary according to a range corresponding to the driver'scurrent viewing range. The observation neglect load begins to becalculated when the vehicle speed is 10 km/h or more in a state in whichthe vehicle starts up. The observation neglect load may be calculated bythe following equation:

$E = {\frac{T_{E}}{T_{{preset}\mspace{14mu}{time}}} \times D_{E}}$

-   -   E=observation neglect load    -   T_(preset time)=preset time    -   T_(E)=time for which driver's viewing range is deviated from        visible range during no-load driving    -   D_(E)=observation neglect load weighting.

As described above, the driver status load may be calculated at S230 byperforming a voice load calculation step S231, a drowsiness loadcalculation step S232, and an observation neglect load calculation stepS233, and then performing a step S234 of summing the respective loads,according the following equation:W _(i) =V+P+E

-   -   W_(i)=driver status load    -   V=voice load    -   P=drowsiness load    -   E=observation neglect load.

In addition, as described above, the driving load may be calculated atS200 by performing the vehicle driving load calculation step S210, thevehicle operation load calculation step S220, and the driver status loadcalculation step S230, and then driving load calculation step S240,according the following equation:W _(total) =W _(d) +W _(M) +W _(i)

-   -   W_(total)=driving load    -   W_(d)=vehicle driving load    -   W_(M)=vehicle operation load    -   W_(i)=driver status load.

The comparison step S300 compares between the driving load of the drivercalculated in the calculation step S200 and a preset load margin. Whenthe driving load is equal to or less than the preset load margin, it isdetermined that the driver is in a safe driving state. On the otherhand, when the driving load exceeds the preset load margin, it isdetermined that the driver is not in the safe driving state. The presetload margin may be an experimental value extracted from a sum of avehicle driving load, a vehicle operation load, and a driver status loadthrough an experiment according to conditions of a test subject. Inaddition, the preset load margin may be a value of the driving loadcalculated based on information according to existing driving patternsof the driver. The preset load margin includes a first load margin, asecond load margin, and a third load margin. The preset load margin maybe stored in the memory portion 3050. The memory portion 3050 may be anonvolatile memory as a storage means for storing data.

As shown in FIG. 47, the warning step S400 includes a first warning stepS410, a second warning step S420, and a third warning step S430. Thewarning step S400 serves to guide safe driving by performing respectivesteps of different warning levels depending on signals transferred fromthe comparison step S300 to inform of a warning corresponding to thedriver status. The first warning step S410 is performed when the drivingload is equal to or greater than a first load margin and less than asecond load margin, and includes a warning sound generation step S411through a speaker, a warning display step S412 through an AVN or a HUD(Head Up Display), and a vibration notification step S413 throughvibration of a steering wheel or a seat. The warning sound generationstep S411 plays an announcement or a warning sound for notifying thatthe driver is not in the safe driving state through the speaker. Thewarning display step S412 displays a warning message or a warning iconfor notifying that the driver is not in the safe driving state throughthe AVN or the HUD. The vibration notification step S413 induces thedriver to have awareness by generating vibration to the steering wheelor the seat. The first warning step S410 is a step of the lowest warninglevel in the warning step S400.

The second warning step S420 is performed when the driving load is equalto or greater than a second load margin and less than a third loadmargin, and holds functions of the AVN. That is, since there is a highpossibility of safe driving being obstructed when the AVN is operatedfor a long time, the second warning step S420 induces the driver toconcentrate on driving of the vehicle by allowing the AVN to notoperate. The second warning step S420 is a step of an intermediatewarning level in the warning step S400.

The third warning step S430 is performed when the driving load is equalto or greater than a third load margin, and is a step of safely stoppingthe vehicle through steering wheel control, transmission control, andbrake control. The third warning step S430 is a step of the highestwarning level in the warning step S400. When it is determined that thedriver may not safely drive the vehicle any more, the third warning stepS430 is a step of stopping the vehicle in a safe region through thesteering wheel control, the transmission control, and the brake controlso as to safely protect the driver.

In accordance with another exemplary embodiment of the presentinvention, when a first warning step S410 is a first warning, the firstwarning step S410 may include a warning sound generation step S411through a speaker, a warning display step S412 through an AVN or a HUD,and a notification step S413 through vibration of a steering wheel or aseat. Since the first warning step S410 is a first warning, the firstwarning step S410 performs a slight warning for informing the driver ofan unsafe driving state. After the first warning step S410 is performed,the process is returned to the information acquisition step S100. Then,the calculation step S200 and the comparison step S300 are performedagain so as to determine whether or not the driver is restored to a safedriving state. When it is determined that the driver is not restored tothe safe driving state despite execution of the first warning step S410,a second warning step S420 is performed. When the second warning stepS420 is a second warning, the second warning step S420 limits functionsof an AVN. That is, when the driver operates the AVN despite executionof the first warning step S410, the second warning step S420 stopsoperation of the AVN to alert the driver to awareness. After the secondwarning step S420 is performed, the information acquisition step S100,the calculation step S200, and the comparison step S300 are performedagain so as to determine whether or not the driver is restored to thesafe driving state. When it is determined that the driver is notrestored to the safe driving state despite execution of the secondwarning step S420, a third warning step S430 is performed. When thethird warning step S430 is a third warning, the third warning step S430safely stops the vehicle through steering wheel control, transmissioncontrol, and brake control. That is, when the driver is not restored tothe safe driving state despite the first and second warnings, thevehicle is autonomously stopped in a safe region against control of thedriver. Consequently, it may possible to protect the driver which is notpersonally restored to the safe driving state.

FIG. 54 is a flowchart schematically illustrating a method of detectinga driver status which includes a driver status determination steputilizing an ECG according to still another embodiment of the presentinvention. FIG. 55 is a detailed flowchart of FIG. 50. FIGS. 56 and 57are views for explaining a method of determining a driver status from adriver's heart distribution chart and heart histogram. Referring toFIGS. 54 to 57, in the present invention, an HRV (Heart RateVariability) is calculated using an ECG measurement value in order toallow a driver to directly analyze a driving load. The HRV is an indexof measurement of a driver's work load, namely, the driving load, and isa method used together with an HR measurement method. Since the HRVobviously exhibits a level of difficulty to which a human body respondsto stimulation, the HRV may be used as a quantified index in measuringthe driving load.

As shown in FIG. 55, driver's ECG and PPG signal information is acquiredthrough an ECG sensor and a PPG sensor as a wearable sensor in aninformation acquisition step S100, an HRV signal is detected from theacquired ECG and PPG signal information in an HRV signal detection stepS510, a heart distribution chart and a heart histogram are derived fromanalysis of a time domain and frequency range of the HRV signal in aheart distribution chart and heart histogram derivation step S520, and adriver status determination step S530 determines whether a driver is ina normal condition or in an abnormal condition through the heartdistribution chart and the heart histogram.

As shown in FIGS. 56 and 57, in the normal condition, it may beidentified that the heart distribution chart is evenly and widelydistributed within a red reference range and the heart histogram forms alarge triangular shape. On the other hand, in the abnormal condition dueto activation of the stress or parasympathetic nerve, it may beidentified that the heart distribution chart is intensively exhibited ata low numerical value and the heart histogram forms a small triangularshape.

When the driver status determination step S530 determines that thedriver is in the abnormal condition, an emergency control step S540 isseparately performed without performing a calculation step S200, acomparison step S300, and a warning step S400. The emergency controlstep S540 may include a window opening step S541, an anion generationstep S542, a music play step S543, a driver warning step S544 through anAVN or a HUD, and a vehicle safety stop step S545 through steering wheelcontrol, transmission control, and brake control. As described above,when it is determined that the driver is in the abnormal condition in adriver status determination step utilizing an ECG S500, it is anemergency situation capable of being a deadly danger to safety of thedriver. Accordingly, the emergency control step S540 is separatelyperformed without performing the calculation step S200, the comparisonstep S300, and the warning step S400.

FIG. 58 is a flowchart schematically illustrating a method of detectinga driver status which includes a driver status determination steputilizing an EEG according to yet another embodiment of the presentinvention. FIGS. 59 and 60 are detailed flowcharts illustrating thedriver status determination step utilizing the EEG. FIG. 61 is a viewfor schematically explaining a method of determining a driver statusutilizing the EEG. FIG. 62 is a table illustrating a frequency range andcharacteristic of each brainwave. FIG. 63 is a diagram for explaining amethod of finding a frequency range for each brainwave using a Bayesiannetwork. FIG. 64 is a conceptual diagram illustrating a driver statusdeduction step using the Bayesian network. Referring to FIGS. 58 to 64,a α wave is increased in a driver's brainwave when a driver has relaxedtension or is drowsy, and a β wave is increased in the driver'sbrainwave when the driver feels tense and anxious. Since a brainwaveactivity may quantify tension and anxiety, the brainwave activity may beused as quantitative data for determination of a driving load.

As shown in FIGS. 59 and 60, a driver status is deduced at S620 byacquiring driver's brainwave information through a wearable sensor of aheadset type S100, separating respective waveforms of the acquireddriver's brainwave for each frequency S610, and finding a frequencyrange for each brainwave using a Bayesian network. That is, as shown inFIGS. 61 to 64, the driver status may be finally deduced using a methodof determining a comparison between the drive status and existing databy indicating an uncertain situation as a probability value through theBayesian network and simplifying a complicated deduction process as arelation between quantitative nodes. The existing data may be stored inthe memory portion 3050.

The driver status deduction step S620 determines whether or not thedriver is in a first drowsy state when the driver is deduced to be in adrowsy state. When it is determined that the driver is in the firstdrowsy state, a first drowsiness warning step S630 is performed. Thefirst drowsiness warning step S630 includes steps, such as a music playor warning sound generation step S621 through a speaker, a warningdisplay step S622 through an AVN or a HUD, and a vibration notificationstep S623 through vibration of a steering wheel or a seat, which arecapable of awakening the driver from the drowsy state. In order toidentify whether or not the driver is awakened from the drowsy stateafter the first drowsiness warning step S630, the informationacquisition step S100, the brainwave separation step S610, and thedriver status deduction step S620 are performed again. When the driverstatus deduction step S620 deduces that the driver is in a second drowsystate despite execution of the first drowsiness warning step S630, theprocess performs a second drowsiness warning step S640 of safelystopping the vehicle through steering wheel control, transmissioncontrol, and brake control. That is, when the driver is restored to theawakened state despite the first and second drowsiness warnings, thevehicle is autonomously stopped in a safe region against control of thedriver. Consequently, it may possible to protect the drowsy driver. Asdescribed above, when it is deduced that the driver is in the drowsystate in a driver status determination step utilizing an EEG S600, it isan emergency situation capable of being a deadly danger to safety of thedriver. Accordingly, the first and second drowsiness warning steps S630and S640 are separately performed without performing a calculation stepS200, a comparison step S300, and a warning step S400.

When the driver status deduction step S620 deduces that the driver is inan anxious state, a system determines an operation condition algorithmfor vehicle driving such that errors are not present in the algorithm,so as to provide the driver with a driving guide through the AVN or theHUD S650. As described above, when it is deduced that the driver is inthe anxious state in the driver status determination step utilizing anEEG S600, it is an emergency situation capable of being a deadly dangerto safety of the driver. Accordingly, the driving guide provision stepS650 is separately performed without performing the calculation stepS200, the comparison step S300, and the warning step S400.

When it is deduced that the driver is in a concentrated or stable statein the driver status deduction step S620, the calculation step S200including calculation of a brainwave load W_(EEG) is performed. Thebrainwave load W_(EEG) is calculated through a signal ratio in theconcentrated or stable state, as in the following equation:

$W_{EEG} = {\frac{\alpha\mspace{11mu}{{wave}\left( {8 \sim {12.99\mspace{14mu}{Hz}}} \right)}}{\beta\mspace{14mu}{{wave}\left( {13 \sim {29.99\mspace{14mu}{Hz}}} \right)}}.}$

When a α wave value becomes a maximum value (12.99 Hz) by dividing amean frequency value of the α wave and β wave extracted for a unit time,the greatest value of the brainwave load W_(EEG) approximates 1.Accordingly, the brainwave load W_(EEG) is maximized. In addition, thecontraposition is established. A value, which multiplies the calculatedvalue of the brainwave load W_(EEG) by a brainwave load correction valueφ calculated by an experiment, is summed in the calculation step S200,as in the following equation:W _(total) =W _(D) +W _(M) +W _(i) +α+W _(EED)

-   -   W_(total)=driving load    -   W_(D)=vehicle driving load    -   W_(M)=vehicle operation load    -   W_(i)=driver status load    -   W_(EED)=brainwave load    -   φ=brainwave load correction value.

FIGS. 65 to 68 are detailed flowcharts illustrating a method ofdetermining a driver status utilizing an ECG and an EEG according to afurther embodiment of the present invention. As shown in FIG. 59, amethod of detecting a driver status according to an exemplary embodimentof the present invention may be performed in order of an informationacquisition step S100, a driver status determination step utilizing anECG S500, a driver status determination step utilizing an EEG S600, acalculation step S200, a comparison step S300, and a warning step S400.A method of detecting a driver status according to another exemplaryembodiment of the present invention may also be performed in reversedorder of a driver status determination step utilizing an ECG S500 and adriver status determination step utilizing an EEG S600. That is, themethod may be performed in order of an information acquisition stepS100, a driver status determination step utilizing an EEG S600, a driverstatus determination step utilizing an ECG S500, a calculation stepS200, a comparison step S300, and a warning step S400.

FIG. 69 is diagrams for describing a basic algorithm of the vehiclesafety support apparatus in accordance with the embodiment of thepresent invention.

As shown in FIG. 69, the vehicle safety support apparatus evaluatescollision hazard level and detects departed driver, and then rescues thedriver from the hazard and performs exit maneuver.

Specifically, in the steps of “Monitor Vehicle Hazard” and “CrashNear?”, outside sensors are used to evaluate collision hazard level andthe apparatus evaluates hazard situations such as leaving lane,approaching other car, unsafe change in speed, etc.

In the steps of “Monitor Driver State” and “Departed?”, driver camera isused to detect driver no longer watching roadway, and steering andcontrols of the driver are monitored to monitor driver no longercontrolling.

And the apparatus requires positive states for both “Driver AppearsDeparted” and “Crash is Very Near” to transition control to autonomous.

In the step of “Transition Control to System”, full control of vehicleis transferred to the apparatus and further driver inputs are filtered.

In the step of “Autonomous Drive”, outside sensors, brake and steeractuators and map are used to rescue the vehicle from immediate hazard,and the apparatus continues to drive and maintain safe behavior incurrent traffic.

In the steps of “Find Safe Place to Stop” and “Safe Place Found?”, mapis used to identify local road side safe area candidates and outsidesensors are used to verify roadside area clear and safe for pullingover.

In the steps of “Exit Roadway & Stop”, outside sensors are used toconfirm clear area off roadway edge, and the apparatus maneuvers thevehicle off of roadway and out of traffic, and the apparatus sendssignal for Help for departed driver.

Although preferred embodiments of the invention have been disclosed forillustrative purposes, those skilled in the art will appreciate thatvarious modifications, additions and substitutions are possible, withoutdeparting from the scope and spirit of the invention as defined in theaccompanying claims.

What is claimed is:
 1. A vehicle safety support apparatus comprising: adriver monitoring sensor configured to monitor a driver; an externalenvironment monitoring sensor configured to monitor an externalenvironment of a vehicle; a driver input filtering unit configured tofilter a vehicle control input from the driver; and at least oneprocessor a control unit connected to the driver monitoring sensor, theexternal environment monitoring sensor, and the driver input filteringunit, the at least one processor configured to: determine criterionbased on data acquired from the driver monitoring sensor and theexternal environment monitoring sensor including multiple drowsy statesof the driver, wherein the acquired data includes that the driver is notavailable before a safe area determination is performed and a hazard isdetected; determine whether to take over a driving control of thevehicle in response to the data acquired from the driver monitoringsensor and the external environment monitoring sensor meeting thecriterion; and perform autonomous driving to move the vehicle to thesafe area in response to determining to take over the driving controlfrom the driver.
 2. The vehicle safety support apparatus of claim 1,wherein at least one processor is configured to: estimate anavailability of the driver based on the data acquired from the drivermonitoring sensor; estimate a traffic hazard based on the data acquiredfrom the external environment monitoring sensor; and determine to takeover the driving control based on the estimated driver availability andtraffic hazard.
 3. The vehicle safety support apparatus of claim 1,wherein the at least one processor determines to take over the drivingcontrol from the driver in response to determining that the vehicle isin immediate hazard situation and that no response is detected from thedriver.
 4. The vehicle safety support apparatus of claim 3, wherein theat least one processor is configured to perform autonomous driving toget out of the immediate hazard situation.
 5. The vehicle safety supportapparatus of claim 1, wherein the at least one processor is configuredto control the driver input filtering unit to block the vehicle controlinput from the driver in response to determining to take over thedriving control from the driver.
 6. The vehicle safety support apparatusof claim 1, wherein the at least one processor is configured to operatea hazard lamp of the vehicle in response to performing the autonomousdriving.
 7. The vehicle safety support apparatus of claim 1, wherein theat least one processor is configured to transmit a signal requesting foremergency assistance through a communication unit, in response toperforming autonomous driving to move the vehicle to the safe area. 8.The vehicle safety support apparatus of claim 1, wherein the at leastone processor performs the autonomous driving based on the data acquiredfrom the external environment monitoring sensor.
 9. The vehicle safetysupport apparatus of claim 1, wherein the driver monitoring sensorcomprises: at least one interior camera for filming the driver; asteering wheel angle sensor; an accelerator pedal sensor; and a brakepedal sensor.
 10. The vehicle safety support apparatus of claim 1,wherein the external environment monitoring sensor unit comprises: atleast one exterior camera, radar, and ultrasonic sensor, configured todetect an outside environment of the vehicle.
 11. A vehicle safetysupport method comprising: monitoring, using sensors, a driver and anexternal environment of a vehicle; estimating, by at least oneprocessor, a driver availability and traffic hazard based on dataacquired from monitoring the driver and the external environment;determining, by the at least one processor, criterion based on theestimated driver availability and traffic hazard including multipledrowsy states of the driver, wherein the criterion includes that thedriver is not available before a safe area determination is performed;determining, by the at least one processor, whether to take over adriving control of the vehicle in response to the estimated driveravailability and traffic hazard meeting the criterion; and performing,by the at least one processor, autonomous driving to move the vehicle tothe safe area, in response to determining to take over the drivingcontrol from the driver.
 12. The vehicle safety support method of claim11, further comprising: blocking, by the at least one processor, avehicle control input from the driver, in response to determining totake over the driving control from the driver.
 13. The vehicle safetysupport method of claim 11, further comprising: transmitting, by the atleast one processor, a signal for requesting an emergency assistancethrough a communication unit, in response to the performing of theautonomous driving to move the vehicle to the safe area.
 14. The vehiclesafety support method of claim 11, wherein the monitoring, using thesensors, of the driver and the external environment comprises:monitoring at least one of physical feature, posture, and controlintention of the driver.
 15. The vehicle safety support method of claim11, wherein the monitoring, using the sensors, of the driver and theexternal environment comprises: monitoring at least one of road andtraffic environment outside of the vehicle.
 16. A vehicle safety supportapparatus comprising: a driver monitoring sensor configured to monitor adriver; an external environment monitoring sensor configured to monitoran external environment of a vehicle; a driver input filtering unitconfigured to filter a vehicle control input from the driver; and atleast one processor a control unit connected to the driver monitoringsensor, the external environment monitoring sensor, and the driver inputfiltering unit, the at least one processor configured to: determinecriterion based on data acquired from the driver monitoring sensor andthe external environment monitoring sensor; determine whether to takeover a driving control of the vehicle in response to the data acquiredfrom the driver monitoring sensor and the external environmentmonitoring sensor meeting the criterion; and perform autonomous drivingto move the vehicle to a safe area in response to determining to takeover the driving control from the driver, wherein the at least oneprocessor is configured to control the driver input filtering unit toblock the vehicle control input from the driver in response todetermining to take over the driving control from the driver.